Scott Bellware

A Simple, Definitive Metric of Software Development Productivity

2011-03-01T08:22:00.007-06:00

Developer productivity is made of two components: 1-) the speed at which work is done when work is getting done, and 2-) the amount of time that is lost when anything stands in the way of getting work done, or more specifically, of getting "value-added" work done.

The most common mistake in measuring software developer productivity is the failure to take both components into consideration.

Productivity is what's left over when you subtract the amount of time lost to non-value-added work from the total time spent working. If you only ever measure your pace when doing value-added work, the results will be a massive over-statement of developer productivity.

Complicating matters further, developers' natural pride in workmanship can make classifying work as "non-value-added" a difficult task. Without an environment that is entirely supportive of transparency, learning, and improving, truthful measurements of productivity are just not possible. Far too many personal preservation mechanisms will get in the way.

The hard, cold truth about developer productivity is that a truthful measure of productivity on a typical software development team is in the 10% to 30% range. Getting a clear picture of the reality of software productivity on your team remains a necessary effort - even in light of the difficulties of undertaking it.

Non-Value-Added Work

Pride in workmanship is what keeps most developers engaged in the work, but it's a double-edged sword. Peter Drucker's famous quote applies: "There is nothing so useless as doing efficiently that which should not be done at all".

We have to come to terms with non-value-added work. We have to consent to it. Non-value-added work simply is.

If we're willing to consent to the reality of non-value-added work, then we'll be open to seeing it. Seeing it is the first step toward measuring productivity meaningfully, which is the first step to developing an action plan for restoring productivity.

If you can get comfortable with the reality of non-value-added work, then you can get comfortable with using its proper name: waste.

Here are a few things that we do that are waste - regardless of any pride we might have in the hard work and perseverance involved:

Teasing apart code to understand how it works, especially when this effort is repeated whenever a team member confronts that code
Performing deep and complex setups of data in an external system (or systems) in order to prove that changes to a local system are correct
Hiding distributed systems behind interfaces that pretend that remote objects are local objects
Failing to create transparency in objects' interfaces as to their underlying latency, or network and storage protocols
Creating development infrastructure, tools, or automation that is only readily-understood by people who have had extensive experience with them
Failing to properly codify or tag tests so that knowledge of which tests to run and how to run them when a change is made is obvious
Designing tests who's failures aren't definitive proof of either defacto functional errors or mistakes in setting up or operating up the test environment
Failing to recognize excess tests as coupling problems that obstruct needed design changes
Designing tests with either excess specificity, test granularity, or penetration into adjacent modules
Underestimating the destructiveness of ambiguity in any code or any other development artifact in general
amongst many other things that can appear as hard-won effort

In short, any work you do that isn't directly related to adding desired capabilities to the system is waste. Any work that you do to resolve obstructions to getting your work done is waste - even if that work improves the overall quality of the system. The reason is simple: obstructions are mistakes that you haven't yet learned not to make.

Some design challenges are due to unforeseen changes in direction in technology or business. As long as you can make those changes without also having to deal with negligent design choices and expediencies of the past, then you're still adding value. The moment you have to deal with obstructions to the work, you're incurring waste - no matter how attached you might be with the designs, mechanisms, and even human organizations and processes that underly the obstructions.

There may always be some waste in the work, but the presence of it doesn't excuse an aversion to seeing it for what it is, accepting that it's there, consenting to its presence, and developing the means to aggressively and progressively eliminate it.

Measuring Productivity

Here's how to get a clear, honest look at software development productivity:

The moment you're forced to divert to non-value-added work, write down the time. The moment you're back on track doing value-added work, write down that time. At the end of the day, add up all the non-value-added time and subtract it from your total work time.

Your work time shouldn't include things that aren't time spent on developer activities unless they're directly related to dealing with an obstruction. For example, a lunch break isn't non-value-added time. However, a walk to the break room to blow off the steam building up from dealing with frustrating design mistakes is counted as non-value-added time. A standup meeting isn't non-value-added time, but a meeting to address an obstruction or to resolve an ambiguity is non-value-added time.

You'll inevitably come across grey areas where some time spent isn't easily classified as non-value-added or value-added. This is especially-true of explicit improvement efforts. Discussing these grey areas with your team and coming to some conclusion is a valuable part of the larger improvement journey that you're on.

Make all the factors feeding your considerations explicit and come to the best decision you can for the circumstances you face. And most of all, understand that different circumstances might lead to different conclusions. Make these differences explicit as well. Open dialog and flexibility is an absolute must, but so is rigor. Keep an open mind (an open heart is also helpful), but be vigilant against allowing a discussion to become a zoo.

If you do a daily standup or status meeting, or have an activity stream app like Campfire, an IRC channel, or even email, report on your productivity every day. Even if you're the only person on your team undertaking the exercise (note: this isn't really sustainable), you can report your productivity.

The goal is to be transparent about the truth of your software development productivity. Once the truth is laid out for all to see, it has a chance of affecting the moment-by-moment decisions that you and your teammates make that continue to confound progress.

Next Steps

Wherever there's waste, there's an accumulation of excess complexity that encourages even more waste. The goal is to fix the design problems that cause it, and to restore productivity to sensible levels. And ultimately, to fix the processes that encourages the waste that exacerbates the design problems that take up long-term residence.

We have to remain utterly unattached to the efforts we invest in solving problems that should not have existed in the first place. If we're not capable of adopting such a disposition, we're incapable of harvesting enough learning about counter-productivity to restore productivity.

If you can measure your productivity truthfully, then you can work with your team to collectively improve the situation, and to continue to improve it.

The start of an improvement effort begins with recognition of a problem and a pathology. Being able to make measurements of the problem is essential to recognizing the problem and also to monitor progress. By developing the ability to recognize waste, you've also given yourself some basic tools to monitor your progress. Measurement might be a bit depressing at first, but it also becomes encouraging as you make progress.

The anatomy of an improvement effort is the subject for another day, but suffice to say that what you're trying to create is a Learning Culture. Doing this isn't a quick-fix, but undertaking it will yield some benefit immediately, and continue to yield a constant stream of value along the way.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Kaizen: "Relentless", Rather than "Continuous"

2011-01-18T08:52:00.005-06:00

"Kaizen" is usually translated from Japanese to English as "Continuous Improvement". It loses its power in the translation.

"Continuous" is perfectly reasonable and correct, but it doesn't really convey Kaizen as the underpinnings of a way doing work. Kaizen is certainly continuous, but it's continuous as a side effect. More importantly than being continuous, Kaizen is relentless.

"Continuous Improvement" can become a flaccid thing - yet another empty corporate mission. At its worst, it reinforces workers' cynicism about cultural initiatives that don't really make work or outcomes any better. "Continuous Improvement" is passive. It's even apologetic and deferential. "Relentless Improvement" is active. It's even fair to say that Kaizen is aggressive.

Often, the real problems robbing an organization of its productivity are the smaller, pervasive problems that are finely-woven into the fabric of the way that work is done and conceptualized. When you solve problems that are inherent in the DNA of the work (and consequently, the organization), you'll likely free up more resources than you might have imagined. A Kaizen culture understands that microscopic, pervasive rot causes a kind of osteoporosis in the bones of the work that can't be seen without actively hunting it with fine optics, or until the bones start to break.

A Kaizen culture doesn't wait for problems to show up. It looks for problems. It's imbued with an awareness that negligible problems often aren't negligible at all - especially when they're pathological and pervasive. It knows that pathological and pervasive problems are often charged with the greatest potential for destruction. Kaizen is literally "looking for trouble", knowing that if it doesn't, trouble will find it. It challenges its belief that these negligible problems are not worth effort or attention. It grounds its exploration in meaningful but practicable, and accessible measurements, rather than in methodology mysticism. In progressively learning what measurements are interesting and practical, it learns to ask better questions.

When a Kaizen culture doesn't see problems, it wonders first whether its vision needs improvement, and sets out to sharpen its eyes rather than rest on its laurels and dull its senses. The adage, "If it ain't broke, don't fix it", is a fallacy. Waiting for problems to come to you means that you're dancing to their tune. The downward, entropic cycle of fire-fighting and decay isn't transformed into a greater capacity to keep moving forward without constantly losing ground.

In the end, "Continuous Improvement" is useful because it's a standard term. But when it's time to actually do Kaizen, "Relentless Improvement" sets more realistic expectations about how it achieves pervasive dominance over pervasive problems.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Testing User Stories - Problem Development Before Solution Development

2010-10-28T08:45:00.032-05:00

User stories can be wrong. Business Analysts and Product Owners can be wrong. They can be the best representatives of user interests but they're often still proxy users. The most expensive mistakes to be made in software product development are made when we get the requirements wrong. A quick way to increases the risk of proceeding to solution development on the basis of incorrect requirements is to not prove that they're right. Or, in other words, to not test them.

The only way to definitively validate that the problem statements for a software product are right is to put the working software in users' hands.

User interviews, focus groups, simulations, and mockups are steps along the way, but their purpose is to reduce the time and cost of getting users in front of an experience that is immersive enough to validate assumptions. Allowing users to experience our proposed solutions to their problems clarifies whether the problem is understood.

Generally, we're taking too long to get stories validated - even in colloquial Agile development. Sending a user story through an entire development process in order to validate it is too much. In order to validate user stories, we need to collapse the solution development timeline so that feedback can be had without the expense of the solution development. We commit to the solution development cost once we've validated the problem.

There are two distinct (but inextricably-intertwined) steps that make up a development process that recognizes the value of reducing the time-to-feedback for testing user stories. The later step is the plain-old Solution Development that we use regardless of whether we recognize a preliminary first step. The earlier step is Problem Development.

Problem Development is software development. There's code. There's deployment. There's testing. There's design and analysis. There's process. There's just much more of it in Solution Development (even Agile development) than in Problem Development. And there are different kinds and variations of these activities in each. Problem Development is software development that works hand-in-hand with business development.

Problem Development and Solution Development have different goals. Problem Development is concerned with feeding Solution Development with input that has less risk (at a time when risk is the riskiest).

Assumptions are liabilities. User stories are assumptions until they're validated. To validate user stories, deliver software experiences that propose to solve the problems that user stories describe, and measure the feedback. There's delivery in Problem Development, but not final delivery.

Final delivery is a concern of Solution Development. Final delivery requires long-term sustainability. Solution Development is concerned with long-term sustainability. When we work toward achieving long-term sustainability before we need it, we end waiting until the longer Solution Development process plays out before getting feedback on user story validity. Which is too long.

Problem Development is concerned with short-term sustainability. The reason that it has to be sustainable at all is that the output of Problem Development is the input to Solution Development, and these two steps are ultimately one process that flows together.

It's far too easy to see Problem Development as "throw-away coding", but it's absolutely not. Problem Development is a way to manage some of the biggest risks in software development - risks that often go un-addressed and are frequently realized as wasted effort and resources, and lower productivity. This kind of risk isn't effectively managed with practices that are satisfied by "throw-away" code.

If throw-away code is the output of Problem Development, then Solution Development has to undergo a (mostly) cold start, benefiting from far less of the momentum created by Problem Development. It might be necessary to make the kind of tradeoff that creates these kinds of batch transfer problems, but making a habit of it is a path to a cumulative productivity deficit.

The kinds of practices that characterize Problem Development might look like software development heresy (or at least "Agile" heresy) to many disciplined solution developers. The realities of Problem Development mean that traditional aspects of Solution Development are short-circuited - after all, Problem Development is looking for a shorter circuit.

The Problem Development practices might even seem irresponsible, but recognize whether you're looking through a Solution Development lens. For just as Problem Development practices can seem irresponsible from a Solution Development perspective, applying Solution Development practices to the point in the timeline best suited to Problem Development can be equally as irresponsible.

Problem Development is about testing or validating user stories' problem statements. Solution Development is about building software based on validated problem statements.

The whole of development process is concerned with finding the way with the least wasted effort in getting from concept to cash.

Feeding un-validated user stories into the Solution Development transformer is often the most efficient way to increase wasted effort.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Myth of Iteration 0

2010-10-25T08:45:00.035-05:00

"Iteration 0" is the initial phase on an Agile project when collaboration tools such as a source code repository server, a continuous integration server, and distributed build and test agents, as well as document tools like wiki servers, and other tools are set up. It's often also a time to configure individual workstations and team members' tools in preparation for everyone starting work.

"Iteration 0" is a bit of a popular misconception that is a side-effect of software projects initiated by technologists - which sounds a bit ridiculous at first. After all, who else but technologists to do software projects? Of course technologists are the right people to do software projects, but are all technologists the right people to execute project startup? And for that matter, what is "project startup" as differentiated from the rest of the work? Is it something that should be approached differently than the rest of the work?

The technology-focused project initiation isn't necessarily the wrong thing to do, but it's often only the right thing to do from within the technologist's perspective, and that perspective can be limited. A tool-biased perspective is a challenging thing to overcome by someone whose moment-by-moment work is necessarily suffused with a constant focus on tools, or whose initial career path passes through a lengthy time of tool focus.

If we literally codify the "first iteration" as "Iteration 1", we technologists can use a bit of geekery to make allowances for a pre-iteration focused on technology. As programmers, we're accustomed to counting a list of things starting from zero. If the list of iterations starts at the "zeroeth" position, but work is only scheduled to start on the first position, then we get an extra "unclaimed" iteration to work with: "Iteration 0".

Any way you cut it, the first iteration is the first iteration, regardless of the numerical designation assigned to it - and regardless of creative license and word games. If you choose to have a zero-based project schedule, then we should naturally change the schedule to state, "Deliver something user-valued in Iteration 0". Which, consequently, is not an invitation to technologists to insert an "Iteration -1" at the head of the schedule.

Over the past few years the caution to not exit any iteration having only shipped technology has been whittled down to a more technologist-friendly caution against exiting an iteration having only shipped frameworks. It's a narrower interpretation of a deeply-powerful principle. It sets the stage for the institution of a technology-focused "Iteration 0".

Nonetheless, it's sometimes necessary to devote a period of work entirely to technology concerns - especially if technology concerns become obstructions to user-valued concerns. And it's all too easy for technology to become an obstruction to delivery in a technology product delivery effort. But this isn't the problem with "Iteration 0".

"Iteration 0" is a project startup issue that is technology-focused startup rather than a product-focused project startup. There's an implicit assumption in "Iteration 0" that - while perfectly reasonable from a technologist's perspective - isn't necessarily an optimal path for project startup.

If a project startup is executed with "all hands on deck", then coordination of the people involved becomes an issue that must be dealt with right of the bat, including the setup of tools that support the coordination.

So, is it necessary to start a project with an amount of people that necessitates coordination and collaboration tools? If project startup is executed with a small workcell of skilled pathfinders, do they need the tools that necessitate the "Iteration 0" work? Can a project startup be executed without the need for the collaboration tool setup?

And more importantly: Is there a benefit to executing project startup without the full compliment of technologists that we'll need for full-steam-ahead solution development?

The answer to this is usually a resounding "Yes", there is a benefit. A good bit of the reasoning and guidance is found more in the Lean Startup and Lean Product Development bodies of knowledge than the colloquial Agile body of knowledge.

There's a time to activate a full compliment of development staff, but the optimal time is rarely the start of development work.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Least Way: Pivoting Away from Agile Excess

2010-10-20T16:00:00.000-05:00

One of the most compelling justifications for Agile development methods and techniques is made through a comparison of the cost curves of both Agile and "traditional", or "non-Agile" methods.

When Agile was new and its case was pleaded more frequently, the following graphic was quite common:

NOTE: The above graphic is intended to represent to relative shape of the cost curves, and not convey exact scales for your organization. Your own organization's cost curve mileage may vary.

The Cost of Traditional Development

The essential proposition is that the cost of a unit of software development effort gets exponentially higher as a project progresses. The same curve represents the cost of change of existing features equally as well as the cost of adding new features.

The underlying principle is pretty straight-forward: the more stuff you have, the more stuff that will go wrong that you can't predict and that you don't detect. The more stuff going wrong, the more rework in-progress. Rework is the crumbly raw material that makes all foundations shaky - be they the foundations of software, or the foundations of the plan.

The number of modules in a software project increases as a project progresses. The number of inter-relationships between modules grows as the number of modules grows. The number of unpredictable repercussions of any change to existing modules or addition of new modules, as well as the extent that repercussions dissipate throughout a system of modules, increases as the number of inter-relationships increases.

When there are fewer modules, it's much more straight-forward to add new modules. However, at a critical turning point, this ease rapidly erodes and it becomes exponentially more difficult to get work done without undoing work that was done previously.

The more modules in a system, the more likely that any new additions or changes will cause adjacent modules to no longer work as they once had, and to no longer work correctly. The longer that they stay in this incorrect state, the more that other adjacent modules will be built incorrectly, or built in a way that reduces development productivity.

The more modules that are in a system, the less likely it is that its programmers can assess any detrimental effects brought about by changes. The system becomes an increasingly unsolvable puzzle that must be solved every time a change is made to add new code or to change existing code. Early on in the life of a system, the puzzle is dead-simple to solve. As the number of modules grow, the possibility of solving that puzzle with every new change closes in on improbable. This is the source of the extra effort that accretes around software development as time goes on.

How Agile Flattens the Traditional Costs

The Agile cost curve is fundamentally different. It was such a radical re-imagining of managing the factors that influence the cost curve that it was frequently dismissed out-of-hand, despite affirmations and reports from people who had given it an honest empirical and experiential trial.

Agile development's power rests in the drive to recognize the conditions that cause the cost of effort to increase over time. It uses techniques that keep effort leveled over the length of a software project, rather than allowing them to spiral toward the inconceivable, unpredictable, and unmanageable.

The cost of change is controllable only when you can predict whether any change will have detrimental consequences. You need to be able to apply countermeasures immediately when any of these detriments are created. The longer the wait to detect a design flaw, the faster the cost grows. This ultimately leads to the vicious circle of rework, where problems are found after the work is delivered to the next phase in the development process, while developers continue to build on that work, believing it to be a sound foundation.

In other words, Agile's radical re-shaping of the traditional cost curve comes from the ability for workers to easily and continuously check that their work isn't a cost magnet while that work is still open on their workbenches. They embrace a principled understanding of the kinds of software designs that are directly and easily tested, and a diligence paid to recognizing, avoiding, and remediating designs that aren't.

And of utmost and paramount concern - concern that can make or break an Agile effort: that organizations realize that its the rigidity of organizations and organizational systems and behaviors that obstruct software development from being done in a way that allows software workers and teams to rehabilitate the traditional cost curve. The impact to organizations - especially entitlements to silos and rigidity - can be profound. Organizations adopting Agile have a responsibility to shift from defensible, garrisoned blame cultures to proactive, optimized learning cultures. If the organizational and management system locks the steering wheel into one, largely immovable position, there should be little surprise when projects end up in the productivity ditch.

The Agile cost curve also suggests that there's a bit of a startup tax to pay up front, but that once the initial bump is navigated, productivity stabilizes and becomes much more predictable.

This is in contrast to traditional methods where the inherent absence of complications early-on lulls developers and managers into a false sense of success. They fail to practice productive countermeasures from the very start by failing to recognize the technical and organizational naivete inherent in the work. The success doesn't last. It's outlived by the realities of the exponentially-increasing number of complications that are natural side-effects of the progress of a software development effort.

The Agile Project Startup Tax

The graphic depicting the Agile cost curve overlaid on the traditional cost curve tells this whole story at a glance. This is what makes it so compelling. And yet, while the Agile cost curve is incredibly compelling, the most interesting point of the graphic isn't so much the benefits that Agile methods avail, but the point on the curve where the lines intersect:

Before that point, Agile development has a higher cost. Before that point, traditional methods should have a lower cost. The time spent using by-the-book, colloquial Agile development techniques before the point where traditional techniques become too costly is a likley volunteer mission for exacerbated costs.

The Best (or Least) of Both

A methodology that uses aspects of traditional development up to the point the costs of traditional methods are on the verge of spiraling out of control, and then makes a methodological "pivot" to techniques that can be considered "Agile" can reap the benefits of each approach's strengths.

Such a "blended" approach takes advantage of traditional methods that, while unsustainable, can deliver more bang for the buck up-front. The resulting path has the least amount of cost if the pivot is well-executed.

The resulting methodology is both traditional and Agile based on where work is in the project timeline and in the smaller development cycles. Neither its traditional aspects nor its Agile aspects are left unchanged by the imperatives of blending what is usually seen as incompatible methods into a continuum. But the most effective and salient principles of development productivity remain in-play.

My own practice of that continuum is informed by Lean Development, and more recently by the Lean Startup movement and its understanding of up-front phases that are often not mentioned by predominating methods like Scrum, and traditional methods that came before.

The Least Way

I was bemoaning failures with Agile and Agile community to Jason Cohen a couple of years ago. He offered the "The Least Way" as a name for the approach that I was embracing to deal with a number of the issues with Agile stasis and orthodoxy that I was trying to work around. I'll ultimately add more meat to those bones on LeastWay.org, and talk about how it underlies some tooling that I'm working on, and how it has affected work on projects. Maybe you'll find it as useful as I have if packaged up in a consumable form.

One of the key differences between an approach that doesn't just cherry pick techniques from different methodologies, but executes a full methodological pivot, is the explicit management of that switch. Planning for it, preparing for it, recognizing the right time to make the pivot, and making a clean entrance and exit from it is a bonafide part of the practice.

Missing the pivot is like missing the last exit before entering the Lake Pontchartrain Causeway. By the time you've realized that you've zoomed past your best chance to take the turn off, you've likely been captured by the gravity well that will start you toward the acceleration of costs. The cost curve begins it's rapid incline immediately after the pivot. The longer it takes to realize that you've missed the turnoff, the faster you'll be traveling, and the more you'll have to spend to put on the breaks and get back on-track toward managing the cost-per-effort.

Shaping the whole methodology around the pivot changes some of the conditions that drive staffing and work management at different phases of a whole software effort. It's not just "changing gears", but more like changing vehicles while they are still moving: you'd want to have vehicles designed for this specific purpose. Modifying the existing vehicles is a good place to start.

With the pivot being a full-fledged part of the methodology, with practices in place to recognize its approach and to execute it cleanly, the cost curve should be smoother, representing the best of what traditional methods have to offer - up to the point that they become unsustainable, and the best of what Agile has to offer - but not before it is sensible.

It's easy to see the methodologies as "one vs. the other" issues - "traditional vs. agile", for example. This perspective can even be helpful when more radical approaches are needed to shake up a rigid, entrenched, silo culture. However, plotting methodologies along a continuum helps us to see those parts of the spectrum where one blends into the other, and to learn to recognize and take advantage of the distinct transitional phases between them that we might otherwise miss or ignore.

Our tendency to look at some colors as "primary" and those in-between as "secondary" is largely an arbitrary side-effect of perceptual bias. The spectrum is one, fluid continuum of material that is neither primary nor secondary. When we choose to see transitional states as secondary, we naturally tend to tune them out. The powerful continuum of abilities that flow smoothly from each other is turned into a checkerboard of disconnected, shallow monotones.

Rapid feedback is the lifeblood of software development that doesn't spiral out of control. Agile protects vital feedback mechanisms from becoming calcified after a projects initial phases. Traditional methods provide rapid - but unsustainable - access to product and customer feedback early-on. Leveraging both and planning for and executing the transition as a fully-fledged part of the methodology rather than a secondary interest is a good part of navigating a "least way" of development obstructions (and costs).

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

User Stories Belong to Everyone

2010-10-04T08:55:00.001-05:00

When Agile development is done well, user stories are always visible to everyone involved in turning ideas into working products. And they're visible all the time. When individuals or individual specializations involved presume a sense of "ownership" over user stories, doing Agile development well becomes difficult if not entirely impossible, save for a veneer of "Agile Theater".

As user stories flow through the whole development process from conception to delivery, they pass through the hands of a number of job functions and workers. Any time that people working in the vicinity of a given work step are given to believe that they "own" the user story in their current purview, they are likely to displace the user story from the most public and most visible medium that is common to the whole team. User stories are often then removed to the bowels of tools that are practical and accessible only to workers at the current work step and its immediate vicinity.

User stories can, do, and should change as they march forward through elaboration from concept to working product. Product development is a process of constantly unearthing a clearer understanding of the work we're doing while we're doing it. No amount of up-front analysis and design can stop this from happening. It's the nature of the kind of work that software development is, ie: product development.

If software development were manufacturing work, we'd know what would be happening in each step of the process before it happened. But then, we'd have to be creating exactly the same product again and again, which would be nothing short of an absurdity for software development.

The further we get into the process of turning an idea into concrete, material product, the greater the clarity and amount of thinking that gets invested into what we're doing. The more thinking invested, the more that we come to understand the subject matter we're working with, the circumstances of product's intended audience, and the fit of the decisions we've made earlier in the process to the reality waiting at the end of the process when real workers will try to do real work with the delivered product.

Analysis and design are present in absolutely every kind of work done in software development, and present in every stage and every work step. And while we may indeed have work steps early in software development processes that appear to be characterized by work exclusively in analysis and design, this is ultimately a consequence of not having concrete product yet in-hand at those early stages.

These early stages are less "Analysis and Design" stages as they are "Absence of Material Product" stages. We resort to seeing early stages as "Analysis and Design" due more to a habit of human perception that tends to draw attention to things that are present than to things that are absent. But later stages are no less "Analysis and Design" stages as a "Testing" stage in any less of a "Development" stage.

It's the same cognitive wrinkle that makes negative space visualizations like Rubin's Vase or the FedEx logo so compelling when we finally recognize the secondary images conveyed by the space that our cognitive facilities filter out. It's also what keeps us from being in a constant state of anxiety due to cognitive overload, and what allows us to be mindful of predators (not to mention to be unmindful of camouflage).

The kinds of work we do in software product development are rarely unique to any given stage. The kinds of work we do accumulates continuously so that later stages have responsibilities for nearly all of the kinds of work done in software development throughout the entire process. That's not to say that front-loading software development work with conceptual work is a bad idea. It's obviously an essential part of doing product development work successfully. But it's easy to mistake early-stage work as the domain of "analysis and design" due to the absence of material product this early in the game. The absence of material product at early stages should be a conspicuous absence, but the conspicuousness is typically lost to cognitive filters.

The disastrous side effect of failing to recognize this negative space perspective is the mistaken conclusion that analysis and design is the sole domain of early-stage work. And it starts the people involved in software work on the slippery slope toward believing that work on requirements analysis only happens early in the process, and that requirements - once defined - will not have to change. As if we could really know anything so conclusive so early in the process of uncovering clarity and understanding.

In the end, early stage work is naturally limited in that it can usually only be conceptual work. Conceptual work is realistically the only work that can be done so early. While the accumulation of responsibilities at work steps later in the process continues, analysis and design never wane. New clarity and new understanding never subsides. In fact, clarity and understanding only get sharper. This continues even once a software product has been made operational.

Because understanding continues to get sharper the further along we are, any concrete record of our understanding must be kept up-to-date to make sure that our collective clarity remains collective. Otherwise, workers at different stages of the whole process will have different understandings of what the goal is - as currently defined by all recent learning.

Making sure that everyone is on the same page is of paramount importance in any development effort. Locking down the definition of what's needed isn't the way to do this - although it's often the mistaken path that software organizations take when they fail to recognize the cumulative nature of understanding and clarity as product development unfolds.

To keep everyone on the same page, user stories must be accessible to everyone involved in a development effort, from early-stage conceptual work, all the way through delivery when placeholders like user stories are replaced with concrete, material product. They must be accessible with the least amount of effort by all people involved and at all stages and work steps.

At any point in the whole process, anyone who is empowered to add user stories, change existing ones, remove user stories, or otherwise make use of them should not have to go through a worker with specialized tool skills due to user stories having been removed to a tool that is not generally-accessible with an absolute minimum of effort and a maximum of immediacy.

Stated as a general principle: No single work group at any work step should remove user stories to a medium that, while more expedient to their work, causes user stories to become less accessible by others with different skills and responsibilities.

The inevitable result of workers at a work step feeling that user stories can be removed to a more locally-optimized medium is that separate copies of a user story will be used by workers at different work steps, and these copies will inevitably diverge. When multiple copies of the same user story diverge, workers at different work steps have different understandings of expectations and goals, and work at cross purposes.

In the worst of these situations, workers at different work steps aren't even aware of the others' divergent versions of the truth, and labor under the resulting conflict without any idea as to why expectations are consistently not being met. This drives up rework, reduces the credibility of the team, increases stress and conflict on the team, and generally leads back to the chaos that Agile development was supposed to address.

And yet, there are indeed advantages to specialized tools for specialized work steps. Great care should be taken though when considering specialized tools for artifacts that aren't specialized. It's often a mistake and an oversight when specialized tools are put in place at a work step that makes collective artifacts less accessible for the duration of the work step, or from that work step onward. In the case of user stories, it's hard to find a single artifact that is more collective, public property and thus less amenable to workstep-specialized tools.

At every stage of development, workers can back-slide into misconceptions of authority over artifacts that are responsible for unifying the whole effort and for stitching diverse workflows and workers together. The further we get into a software process, the more likely it is that workers at a later stage will remove user stories from more generally-accessible media and sequester them into the bowels of tools that are accessible only to people with specialized skills.

For example, consider the Cucumber framework and tools developed in the Ruby community, and the family of derivatives and clones that have since been created in a host of programming languages.

This style of tool encourages the displacement of user stories from more generally-accessible media into programmer-specific media. It doesn't do so accidentally. It's done as a recommended practice - reinforced by articles, books, and software conference presentations and workshops.

While Cucumber is built on some fairly compelling technology, and is itself an impressive work, it's not exactly methodologically-sound in that it fails to recognize its own contradiction of Agile development's core value of individuals and interactions over processes and tools.

As an aside, I brought up this issue with Aslak Hellesøy, the Principal of the Cucumber project, during a trip to Sweden last year. At the time, he informed me that work was being done on a tool to extract user stories from the specialized media to make them generally-accessible. But there are three non-trivial oversights inherent with this solution:

Firstly, it disregards the a-priory value of whole-team organizational methodologies like Agile by allowing collective and communal artifacts that are fundamental to higher-order productivity to become appropriated by one specialization.

Secondly, it seeks to address a problem that exists because of overly-elaborate tooling by creating even more elaborate tooling. The path back to simplicity and clarity - i.e.: "elegance" - likely also suggests removing the first wave of tooling rather than adding a second wave of compensating tooling. There are presently solutions that are arguably more holistically-effective as well as arguably more simple and clear. Although, they're based on less elaborate programming technology, and this is likely why the programming specialization remains distracted for alternatives.

Thirdly, it doesn't address the need to allow user stories to be added, removed, combined, divided, or changed by anyone with the authority to do so regardless of whether he or she is comfortable with tools, approaches, and perspectives that are more natural to the programmer specialization.

A Cucumber-style tool can be justified arguably as a general tool because it serves the whole of the process with testing and specification, but merely serving the whole of the process doesn't atone for the reduction in accessibility and immediacy of collective artifacts.

To Aslak's credit, he demonstrated a way of using Cucumber to me where user stories are not removed to its media. Nonetheless, removing user stories to Cucumber media continues to be a practice that is perpetuated in the user community remains a non-trivial methodological problem that is likely rooted in the narrowing of focus of specialized workers due to impressive tooling.

NOTE: There are more methodological oversights inherent in these tools, but that's a subject for a subsequent article.

Here a some principles to keep in the forefront of your Agile practice amidst the distraction of so many Agile-targeted tools:

1. Individuals and interactions over processes and tools has profound meaning. It doesn't suggest that you shouldn't use tools or processes, but that these often become distractions from more powerful means of ensuring success. Consider tooling choices very carefully - especially specialized tooling. Understand how it might inadvertently narrow the perspectives and values of its users.

2. User stories are a whole team artifact. Any tool that moves user stories to a medium that the whole team doesn't have at their finger tips should only be used when there are no viable alternatives. Forcing some team members to go through other team members to work with collective artifacts is categorically not what is intended by individuals and interactions over processes and tools - even if it does cause people to work together to gain access to resources. These resources are supposed to be generally and immediately accessible to begin with.

3. If you choose to remove collective artifacts to workstep-specific tools, put protocols in-place to ensure discrete hand-offs between tools so that multiple versions of the truth are not available to different workers who work in different work steps. When a user story moves from one system to another, destroy the previous version and leave behind a placeholder - a kind of forwarding address for a user story - that tells the interested party how to find the user story's new home.

4. Don't fool yourself into believing that you'll be able to keep multiple versions of user stories in-sync through the life of a project. This kind of work is too costly to keep up and will be quickly abandoned by the project team, leading straight back to the multiple versions of the truth problems. Adding customized electronic synchronization automation to this problem is also not a good answer. This just adds more elaborateness to a situation that is possibly already too inelegant. It's a problem that likely shouldn't exist in the first place.

The inescapable truth of user story management is that the least-elaborate technology is often the most productive solution - even if the technology is no more elaborate than a ten-foot by five-foot cork board with index cards pinned to it. In the very least, this keeps user stories immediately visible and accessible, and makes changing them no more difficult than putting pen to paper.

Of course, the tried and true cork board and index card solution isn't a good foot for all teams. That said, as you seek more elaborate solutions, take tiny steps forward in settling on your solution. A leaping at the most elaborate solution for user story management usually ends up planting a whole team in a tool that serves the specialized perspective of the people doing the tool selection - often to the disservice of everyone else involved in the whole effort.

It's vital that we're not lured into local optima interpretations like, "individuals and interactions over tools and processes except when the tools are intellectually stimulating". All tools are intellectually stimulating. In many cases, tools can be stimulating to distraction. The problem is what we're being distracted from: the essential root cause of success in software product development. Namely, the magnification of understanding and clarity that comes from interactions between individuals over the course of time. The deeper problem is that we believe ourselves to be impervious to this kind of distraction because we have such positive feelings about our tool choices.

The immediacy and accessibility of user stories is a foundational corner stone that your Agile house is built on. Whether it stands strong and continues to be built upon, or whether it crumbles under its own weight can be decided entirely by this salient factor.

Do everything you can to make sure that user stories continue to belong to everyone, in every minute of every day. And be vigilant against any backsliding into specialized expediencies that remove user stories to any medium that is less immediate or less accessible than the simplest and clearest of tools at our disposal. An elegant solution is often not an elaborate one, but greater whole-team productivity is always more stimulating than elaborate, specialized tools.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

User Stories are Temporary

2010-09-28T09:55:00.003-05:00

It's obvious, but warrants mention: What we do in the future is likely to be different from what we're doing today. The implications for user stories should be obvious: User stories are temporary. Saving them for posterity doesn't serve the primary purpose of user stories, and doing anything that makes them less temporary can turn user stories from benefit to detriment.

User stories are a bit of semi-refined raw material that systems are created from. They're a way to represent users' needs. Most importantly, they communicate an understanding of users' context to the people who are building the software. User stories help to make it more difficult for software development teams to succumb to the ever-present threat of losing track of the user's bigger picture.

Aside from raw human ideas that user stories represent, they are the most flexible artifact in software development. Then can be created at any time. They can be changed at will. They can be combined. They can be subdivided. They're not intended to become fixed, and anything that serves to make them fixed is a disservice to the value they offer. User stories aren't fixed in a point in time, just as needs aren't fixed. They change because conditions change. They change because our own understandings change and improve as we gain the clarity that comes from working with user stories, and from building the products that satisfy needs.

When we realize new clarity, our responsibility is to reflect that new clarity in the form of new stories, or new subdivisions of existing user stories, or through the combination of existing user stories, and even from the removal of user stories from the product definition entirely.

The user stories that describe our needs tomorrow are going to be different from today's. Knowing this as we do, we have we gone from recognizing the transient, temporary nature of user stories to the point where we now tend to capture them in tools that make them harder and to change, and harder to take in at a glance and to glean new clarity from. When we do this, we're increasingly unlikely to update user stories to reflect the improved understand we cultivate as we work with them, with users, and with the software that we're developing.

The more that we calcify user stories in ever more concrete media, the less we make necessary course corrections along the way. There is little that so obviously contradicts "Agile" more than inviting anything that adds to the calcification of requirements. A principle duty of user story management in Agile development is to protect them from becoming encumbered by the notions of permanence that obstructed software development methodologies that predate the effectiveness of Agile methods.

We pat ourselves on the back for having evolved beyond the quintessentially-primitive "300-page requirements document", but often all we're doing is bringing some of the same old problems forward to our "new" requirements formats. Sure, we've outgrown the massive, big-batch requirements doc, but the genetic material of the disease that infected those documents have evolved into a newer problem for newer artifacts, and ultimately, the problem is still the same essential problem. We're so pleased with ourselves for our new Agile ways that we fail to see that we're repeating the same mistakes, except with new material.

Here are some things to keep in mind that can help with the relapse into the same old mistakes:

1. When you forget why you're doing something the way you do, the answer isn't in a user story archive. The answer is in revisiting what you're doing to make sure that it still holds true. Do this with interactions and with people. Use tools and processes to support the human interactions.

2. User stories are not artifacts for regulatory compliance. User stories are a scratch pad of human understanding at the present moment. They are chuck for the software development grinder. What comes out the other end should be the software that itself explains why things are as they are. If you need a permanent record for the purposes of regulatory compliance, go ahead and create one. Nothing in Agile stops you from doing that. Just don't hang a stasis anchor around the neck of user stories. If you do, you'll strip them of the benefit that they bring to software delivery.

3. Have one master copy of a user story while that user story is in-progress. The master copy should be the one that is most immediately accessible to the most people who are involved in turning ideas into software products. They should be be posted using the most visible medium possible - one that requires the least amount of human interaction to casually-scan a project's gallery of user stories.

4. Throw user stories away once you deliver the working software that addresses the needs expressed by user stories. The software and its supports are enough to describe why it is the way it is. If you no longer remember, and if the software and its supports and the mechanisms and practices of institutional memory aren't sufficiently self-descriptive, revisit all of these issues as an opportunity to improve the production system in a way that doesn't invite more tool-centric bureaucracy.

Here's a bit of a side note to consider: The principle sponsors of the Agile 2010 conference are both "Agile" tool vendors.

If one of Agile development's primary tenants is to value people and interactions over processes and tools, how is it that the only Agile organizations producing enough disposable income to support Agile development itself are tool and process companies specializing in Agile requirements management and requirements compliance? Somehow, we've walked in a circle - yet again. And one of the things we've left on the trail behind us is the understanding of how things went wrong last time, as well as our commitment to avoid expediencies that don't serve sustainability.

The process methodologies that predate Agile are often roundly dismissed by agilists. But there wasn't really inherently wrong with them. Things back then started getting silly when tools started getting silly; when we began to defer to tools as authorities rather than to the people needing and the people solving needs.

One of the first definitive steps into the downward spiral of mindless process bureaucracy hell is the calcification of requirements. And no matter how temping it is to succumb to the materialism of hardened collections of user stories, we don't get to hell without passing through those gates.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Workplace Safety for Software Developers

2010-09-13T10:00:00.010-05:00

Software development managers make sure that their workers' physical safety is secured - assuring that office ergonomics are up-to-spec, and in compliance with the prevailing laws and regulations. On the other hand, managers verily require that software development workers undertake some of the riskiest behavior in the contemporary workplace. It's entirely common that software managers require workers to undertake work the crosses the line of being merely questionable, and moves squarely into the territory of negligence. It's not because software development managers are nefarious, but because they are too-often far enough removed from the work of software development to differentiate negligent and hazardous decisions from safe and sound decisions.

Without a tangible understanding of the work undertaken by software workers on a day-to-day basis, and an understanding of that work in great detail, software managers are not equipped to know when they are requiring software workers to supplant safety with expediency.

Workplace safety is an issue of productivity. Lack of safety breeds low productivity. In the case of software development, "safety" is an issue of undertaking work in a fashion that doesn't expose the software worker to unnecessary risk - especially risk that is easily avoided. That is, it's easily avoided for someone who understands the work of software development at a high-enough level of expertise and detail to guide workers away from unsafe practices.

An unsafe practice in software development is any work done in the present moment that makes work in the next moment take longer to complete. A reasonable analogy for this is the notion in physical materials work of a clean workplace.

The purpose of a clean workplace is to remove the clutter that hides unseen, unsafe conditions. It's a lot easier to trip over an errant piece of material in the workplace when its surrounded by a profusion of other errant materials. When hazards are so common that they blend into the background, we simply stop noticing them, and this is how we come to be put at greater risk.

The more unsafe clutter we have in our workplace, the more we have to work around. The more we have to work around, the more time it takes us to do work.

If a manager can't detect the conditions that makes work take longer to accomplish than expected, he doesn't adjust his expectations for cycle time accordingly. This means that he expects more to get done - the original work, plus the workaround work - in an amount of time that would be reasonable for the completion of the original work alone.

This makes workers even more careless, exacerbating the accumulation of obstructions and hazards in the software development workplace as a result. It's a vicious cycle that usually only happens because someone with the authority to avoid problems in the first place doesn't have the experience to recognize the problems and to deal with them.

In the words of the ancient Chinese workflow master, Lao Tzu: "Confront the difficult while it is still easy. Accomplish the great task by a series of small acts."

Although, you can only do those small tasks if you can detect their presence. The finer your ability to perceive counter-productive deviations, the quicker you can respond to them. The longer you wait, the further off-course you'll be when you finally realize that you need to make a course correction. Someone who doesn't do work can't recognize when that work is off-course. Someone who isn't an expert in the work can recognize when the work is off-course, but his will only recognize it after it has become more expensive to deal with than necessary.

And this, as the average software worker would tell you, is the day-to-day conditions in which their work is done. Software work is a constant battle fought against the accumulation of hazardous clutter. Software workers rarely get ahead of the clutter curve, and have to invest significant effort to keep their workplace free of hazards. Paraphrasing Taiichi Ohno, if you're not moving forward, you're falling behind.

Each individual scrap of hazardous material in software development is typically quite small - small enough in fact to be easily deemed negligible - even by people who do the work. The hazard presented by two iota of hazardous waste in software isn't the sum of the hazards - it's the sum of the hazards compounded by some multiplier. Hazards in software don't exist in isolation, they interact with each other creating a higher order of hazards that is greater than the sum of their parts. Two pieces of hazardous software that interact with each other don't create two hazards, they create two hazards compounded by the amount of interactions between these modules. To make matters worse, all modules that interact with hazardous modules in close adjacency also get infected. Lack of day-to-day expertise in software work often leads to negligent underestimation of the risks associated with the "design hernias".

Software hazards compound very quickly. It's very easy to arrive at an explosive accumulation of hazardous software materials. It can happen within the first few days of a software project. If subtle hazards in the tools and frameworks that programmers are slated to use are not recognized immediately, the accumulation begins before the first line of code is even written.

When we ask software workers to continue to work in such conditions, we might as well send them into a mine filled with coal dust and require them to hyperventilate for several hours each day. Sooner or later they are going to have to escape the job to escape the hazards, or they are going to acquiesce to the irreconcilable differences between a manager's expectations and the realities of the working conditions that the manager himself feels entitled to not be exposed to for having already "paid his dues". This creates dispassionate cynicism in software developers. They come to learn that no matter what they do until they escape the work, they are more than likely going to end up on the losing end of the software work proposition. They learn that they'll get hung with the detritus of negligent software development decisions and direction because decision makers are too far removed from the work to see that it is these critical decisions and directions (or the lack thereof) that are the root cause.

In these working conditions, workers constantly face obstructions that are only in-place because they weren't recognized when they were mere seeds of problems rather than towering oaks of looming counter-productivity.

The accretions of hazards into founts of the depressed productivity that is status quo for most software teams is fully and completely avoidable - if only the hazards are dealt with when they are small. To see looming hazards when they're small, you have to have detailed knowledge of the work in the here and now. This is simply not possible when managers have removed themselves from the work.

It's far too common for software development managers to feel entitled to be removed from the work of software development - as if removal from the work is a reward for having done the work for a number of years. The reward for doing software development work for a number of years is not an escape from the work, but an immersion into a far deeper understanding of it so that its expert insight can guide software work away from hazards. Of all the escapes that can be orchestrated by software workers, an escape into management is indeed and in fact an act of pure negligence.

Software work is works in intangibles. Software hazards go unnoticed because they are physically invisible. Taking a bit of liberty with Donald Reinertsen in The Principles of Product Development Flow, we don't see the manifestation of software development safety issues "when we walk through the engineering department." They're "bits on a hard drive, and we have very big hard drives in product development."

Managers often try to compensate for their lack of detailed understanding through the use of summary representations like software diagrams. But software diagrams don't show the looming problems while they are still small; still manageable gathering storms that can be dissipated through judicious application of minimally-invasive countermeasures. Only flagrant hazards can be detected in summary representations. Small, detailed course corrections can't be plotted from coarse, summary information. And yet managers who feel entitled to be removed from the work perpetuate the fantasy that summary representations like diagrams are sufficient to bring their purview into action on the software projects they manage. This is pure folly, and software tool vendors are quite happy to continue to exploit it.

By the time you've detected a software hazard that can be seen in a summary representation like a software diagram, you're looking at a problem that has festered for far too long. You should be able to detect the chemical markers of the disease long before you notice that lump in a vital organ.

Programming work is almost entirely mental. Its effectiveness is influenced by psychology, cognition, awareness, and communication far more than by any material concern like ergonomics, an ultra-fast workstation, or multiple monitors. Software systems are far too large and far too complex to be held in entirety in the conscious focus of any single software worker in any single moment. The devil is always in the details. Summary representations are secondary to the actual raw materials of software: the code. You have to be in the code, and to know enough about code to understand which subtle design differences are looming problems and which aren't.

The average software system is a dark coal mine filled with the poison particulate of tomorrow's case of black lung. We ask developers to work in these conditions every day. We do this because at some point in our careers as managers, we believed that we were entitled to be removed from the details of software work.

Sooner or later, enough hazardous software material accretes in a software system that managers step in, and often the first thing they do is look for a root cause in the workmanship of the software workers. The workmanship is rarely the root cause. The absence of informed, skilled, and insightful software management and guidance is always a preeminent cause to unattended workmanship. The workmanship is, of course, a problem, but it's a side-effect.

This is like blaming miners for an underground explosion due to the accumulation of day-to-day hazards that result from institutionalized negligence of clear and present dangers that should be managed the moment that they show their first signs. We ask workers to undertake work that they know is not in their best interest. We ask them to do this from our organizational perches far about the hazardous conditions of software projects. And then we hang them with the inevitable costs of this kind of mismanagement.

It's the software worker that is required to spend extra time in the mine to balance productivity that is lost to management negligence. It's the software worker who is required to take on the duties on sanitizing data by hand without the safety of the proven automation that should have been built right to begin with. We ask them to validate the decisions that we've made about the tools that they'll use in their work based on little more than a compelling sales pitch from a tool vendor who claims to knows more about the day-to-day work of the software developers in our organizations than we do - vendors who are even further removed from the work than we are.

We ask software developers to work against their own interests and the interest of our organizations because we don't understand the hazards that our decisions create.

Software workers suffer disruptions to their lives and their livelihoods when this kind of institutionalized negligence drives work policy on software projects. They shoulder the accumulated detritus of uninformed and unskilled management decisions. And they suffer the humiliation of blame when lack of management insight can't see it's own reflection in the root cause mirror. And institutions lose the invaluable institutional knowledge when workers escape the organization altogether. It's always a no-win situation. Everyone loses. And it's completely avoidable.

If you're not willing to be in the day-to-day work of software development, you're declaring loud and clear that you're not qualifying yourself for the authority to make decisions that direct software work. Every decision you make risks adding to the accumulation of software hazards. When you do this, you deplete the workplace safety for software workers. You ask them to take risks that the workers themselves know that you can't see, and they know that because you can't see them, that you will likely not understand that the accumulation of avoidable hazards into full-fledged, clear and present dangers is your fault to begin with.

Workplace safety is a serious productivity issue. It's a serious issue for the health of workers. It's a basic expression of human respect for the people who work for software development organizations. Understanding workplace safety for software developers requires a high level of expertise in software development, and it requires day-to-day currency in software development. A manager who is not willing to have insight into the details of the work that he is responsible for is patently disrespectful of his workers. He constantly puts them into harm's way by presuming to express authority without knowing whether his expectations are hazardous to the health, well-being, and viability of software work and software workers. Beyond disrespectful, it's dishonorable.

The software field isn't at risk of programmers forming a united front and leveraging collective bargaining, and frankly such a thing isn't likely what anyone wants. But dealing with the root cause issues that have driven other industries to such actions is a win for everyone when obstructions to productivity are removed in the process. Addressing software development workplace safety is a good place to start - especially in our current economic conditions, where a massive boost of unimaginable untapped productivity would be a welcome change indeed.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Unfortunate Case of Agile Coaching and Servant Leadership

2010-09-10T08:40:00.041-05:00

The need for coaches, and especially servant leaders, is a side-effect of an unfortunate organizational pathology.

Consider this quote from James P. Womack: "We've got too little management and therefore need too much leadership".

Someone who has become specialized as an Agile coach necessarily benefits from the perpetuation of Agile coaching. This doesn't mean that Agile coaches are nefarious and interested in suboptimal outcomes, but perpetuated specialization typically creates narrowed perspectives and thus overly-narrow interpretations of conditions and needs.

The result is often the very same counter-productivity that we observe when someone is given the exclusive job of capturing requirements for a software project: they fill all of their time exercising their specialization, creating an inventory of work that is out of sync with the progress of the team.

When we bring Agile coaches into software development efforts, we should recognize the underlying problem that is being signaled: that software development managers are not presently equipped to manage the software development work that they've been tasked to manage.

By failing to recognize the underlying problem, we often go too lightly into engagements with Agile coaches, and we end up failing to use them to solve the root cause problem, and this perpetuates the need for longer engagements with Agile coaches. This, in turn, drives Agile coaches further into the reaches of specialization, and narrower perspectives, accelerating the erosion of their usefulness.

To break this cycle, we need to recognize the most important reason to bring on an Agile coach: The purpose of an Agile coach is to transform managers and the management system so that managers, in turn, can transform their organizations.

Coaches initially take on responsibilities that ultimately need to be the responsibilities of a manager. Even if the outcome is an improvement, when a manager's job is split between an authority position (the Boss), and a guidance position (the Coach), the outcome isn't as good as it can and should be. Nonetheless, if a manager is in a position to need a coach, then the coach is inevitably going to take on some of the manager's responsibilities for teaching and transformation while the manager undergoes rehabilitation of his own.

The coach and the manager must recognize that these conditions are a necessary evil. The goal is a transition away from the split direction of a team between a manager and a coach. The longer that this split is in place, the higher the likelihood that the new behaviors learned by the team will be undone by the manager when the coach moves on.

Agile coaches can find themselves as removed from the work of software development as the managers that they are called to rehabilitate in the work. For an Agile software development coach to remain effective, he has to be an active software developer. Just as managers fall into the trap of losing software development currency, so do Agile coaches. An Agile coach who doesn't also do software product development for a living is a ticking timebomb of counter-productive guidance for a software development team.

Consider this quote from Daniel Markham: "If you're going to train something, you should be able to do it. And I mean do it to a very high level of expertise. An agile coach should be able to code, perform analysis, manage the project, test -- anything that needs doing on a project. If she can't, then how can you talk to her about your particular situation? If your agile trainer was a BA last week, or never slung code in his life, or is a professional trainer, or -- let's be brutally honest -- is making less than the members of the team are, you've got a dud. It seems like common sense but it bears repeating: you can't train something you haven't done. And "done" means a bunch of times, not just on the pilot project."

The qualities and abilities that Daniel outlines are qualities and abilities that we need to expect from software development managers. If they don't have those qualities and abilities, they need to get them, and a coach might be a way to get there. If the coach doesn't have those qualities and abilities, most of what's produced during the engagement doesn't become lasting, transformative value.

Agile coaches who themselves don't know the work of software development can't teach managers to know the work sufficiently to manage it. They can only teach them to become other disconnected Agile coaches. This is how the second half of the first decade of Agile came to be dominated by the doctrine of Servant Leadership. As more senior leaders in the Agile community spend more and more time in roles that are ever distant from the work of software development, Agile itself is becoming more robotic, more mindless, more of an orthodoxy, and even more of a liability than a benefit.

Servant leadership is one approach - one tool - fit for the organizational conditions that it serves best. An Agile coach often embraces the edicts of servant leadership as a universal panacea when his duty to stay sharp in the full curriculum of software development is dulled by the ease of Agile coaching specialization. But any tool used indiscriminately is going to undo any benefits that it provided in the circumstances where it was an appropriate choice.

A servant leader's typical proposition is that an organization's ability to meet expectations will be transformed if only its inherent creativity can be tapped and unleashed. While tapping and unleashing unused creativity is transformative, it's extremely rare that doing so is either sufficient or sustainable. It often creates conditions that not only lead to unmethodical exploration of ideas, but also makes teams feel that they have a right to this kind of behavior. In other words: "self-determination" rather than "self-organization".

Servant Leadership as a panacea is also a real side-effect of past experiences with bully management. It's an over-compensation reaction to some very real and very tangible painful experiences of abusive directed authority. It's an equal and opposite reaction to an extreme. It, as well, is an extreme. It's as much an undesirable specialization of behavior as abusive directed authority. It's often a programmed predisposition, rather than a conscious choice informed by circumstance.

The Servant Leadership doctrine often patently dismisses the greater goal of balance between directed authority and creative enablement, almost entirely eschewing the value of informed directed authority and skilled directed authority in favor of an over-indulged creative enablement.

A job in creative enablement is a lot of fun, and many Agile coaches have come to feel entitled to these kinds of engagements. They're light, they're fluffy, they can come with no measurable expectations, and they're often damned lucrative - especially in software organizations in the thick of the detritus of an unrecognized management crisis.

The increasing prevalence of Servant Leadership also presents a great opportunity for consultants with little real experience in Agile to re-brand themselves as Agile consultants. It allows for anyone with even a modicum of summary knowledge of Agile development to have a role in the Agile bonanza. But we all lose when this happens.

If you don't already know what your problem is, then pre-supposing an Agile(tm) solution further deepens the mindlessness of robotic management, and it diminishes the Agile body of knowledge. And ultimately, it exposes our organizations to the risks of bringing in help that may only have the ability to talk the talk - which is often all that is required of Servant Leadership - and often all that servant leaders assert is required of them.

Again from Daniel Markham: "One guy (famous author again) basically put it like this to me when I told him the team wasn't succeeding: I'm here to demonstrate certain practices and to show that they work, not to just stop everything and attend to what the team is dealing with today."

To move forward through this crisis, we need to set new goals for software development management: To be close enough to the work to be able to manage work, and to use directed authority in service of helping the team meet expectations when creative enablement turns "self-organization" into entitlements for excessive experimentation.

And we need to re-define the role of coaches from servant leaders to temporary co-managers, helping the team to meet expectations while bully management and its detritus is transformed into effective software development management.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Manager or Bully?

2010-09-08T08:40:00.009-05:00

Management in the large: managers set expectations for their workers. Of course there's more to it, but "setting expectations" covers a lot of ground.

Management is central to the productivity crisis we face in software development. It's a crisis that pervades software development so profoundly that we usually don't recognize it. It's like water to a fish - not something that our attention is drawn to, not something that we notice. It's the environment that encloses everything we do. It's not that we see it as normal, we just fail to see it at all. Frustration, burnouts, blown budgets, missed expectations, and dissatisfied customers; it's just the way that most software development is done.

We start with the best of intentions, and despite our intentions, we usually end up in the weeds. We end up in the weeds most of the time because that is where software development teams are directed. It's because of one salient, painful truth about software development:

Software development managers don't understand the work of software development sufficiently to be able to direct the work of software development, and the value of hands-on work experience they've had in the past decays so rapidly that it's increasingly unsafe to make new decisions based on it.

Said more plainly: at a time when senior staff should be best informed by their lengthy and varied experience, they remove themselves from the work of software development long enough to lose the understanding of how day-to-day software development work is done. They lose the knowledge needed to direct the work of day-to-day software development. A certain egoic mania takes root that leads senior staff to feel entitled to believe that their insight has moved beyond the necessity of mere practical work experience.

How long does it take to become unqualified to set expectations for software development work? Not long. It happens as quickly as six months and often not longer than two years. This is why the proposition of Self-Directed Work Teams that has been popularized by new, opportunistic interpretation of Agile Methods has become so attractive in software development - even though SDWT is a well-known failure mode in most development industries and is known to be sustainable for no longer than a couple of years in the majority of cases (there are exceptions, but exceptional organizations usually aren't the rule for unexceptional organizations, despite our best hopes and ideals).

Plotting software development management on a continuum between the extremes of Managers and Bullies, we don't find many managers who are only either extreme. A software development authority is rarely either only a manager or only a bully, but software development managers are usually weighted more toward the bully end of the spectrum. It's an innevitable consequence of having lost interest in the details of software development work, and having become quickly dated as a consequence.

Someone who sets expectations but who isn't able to show workers how to fulfill expectations isn't managing, he's using the implicit threat of crossing an authority to make demands without knowing if those demands can be met. And he's hoping for the best. But we need more than hope to manage software development. We need to know that the work being done every day is the work that leads to the desired outcomes.

Setting expectations without knowing how to meet those expectations almost always leads to unqualified or unrealistic expectations. Using authority to require unrealistic expectations to be met always creates a compounding accumulation of obstructions to subsequent work. It's a sure-fire path to lower productivity. Most software development organizations live perpetually under this cloud of depressed productivity.

A bully sets expectations for workers but doesn't have the relevant and necessary capacities to help those workers fulfill the expectations. A bully has been disconnected from work for long enough that he (or she) is no longer able to personally act to ensure that his workers meet his expectations - to show them how the work is done, and to qualify the work that they do.

In other words, the person who needs to be the most experienced and most expert software development worker on the team is often the least capable.

A direct manager knows how the work should be done. He may know this better than his workers. When necessary, a manager can sit down with the worker's current tools, and teach the worker how to meet expectations, and to teach the worker how the specifics and tactics are tied to a greater whole. A bully hides behind assertions that his time and intellect are too valuable to be spent on such trivialities as the "how" of doing the work in his purview.

It's fair for a person with authority to no longer have the capacity to ensure that workers are materially able to meet expectations, but then that authority must defer to someone who does have the necessary and relevant capacity. And this is an entirely undesirable circumstance. It might be the circumstance that you're dealing with, but it's not the one you should settle for.

An authority who isn't capable of knowing whether and how work can be accomplished and whether expectations can be met is qualified to set only summary expectations. An authority who over-steps the material bounds of his capacity will continue to inject productivity obstructions into his organization's workflow by causing workers to make unsafe and counterproductive decisions that innevitably lead to rework, overwork, and relearning.

A bully believes that his workers are soft and lazy, and often puts extra pressure on workers believing that this is what it takes to get merely nominal productivity. A manager recognizes that workers who retreat from work are likely facing some distressing obstructions in the workflow. He uses his skills as a teacher and his insights into the work as an expert worker to help his workers to recognize and clear obstructions.

It's often hard for us to accept the cold reality of the depressed productivity in our organizations. We fail to manage software development as a product development discipline and tend to manage it as a manufacturing discipline (usually without even understanding or recognizing the difference), undermining productivity at every turn by tuning the wrong variables and turning the wrong organizational and procedural knobs.

In an environment where management leans more to the bully side of the continuum, we see work teams stretch the intentions of "self-organization" into the realm of "self-determination" or "self-direction". It's not necessarily an unwarranted reaction, but its a reaction to a root cause that should be the focus of organizational and procedural improvement. It's in these environments that we also see a need for coaches to fill in the management void, but this is also a lesser palliative that doesn't necessarily get to the heart of the matter.

A manager is someone who balances qualified and expert worksmanship with the ability to teach and is comfortable with using directed authority when necessary. He goes with his team into new learning and new understanding, and re-directs the team if the creative drive for experimentation starts to occlude the imperatives of delivering material product.

Doubtlessly, it's sensationalistic to divvy up the software development management landscape into extremes, but doing so is meant to frame the issue so that we can begin to recognize a real problem and begin to do something about it.

The way that software development is practiced at large, it's perfectly understandable for software development workers to become burned out on software development, and to look for a change. Unfortunately, moving upward to software development management isn't where people who are burned out on this work should go. It's little more than an escape, and it needs to be more of a deeper commitment to the work rather than a means to justify avoiding the details.

Ultimately, software development work will remain palatable for much longer when it is managed much better. But we are where we are, and the general population of software development managers will remain skewed heavily toward refugees from the work of software development for some time.

This is the software management crisis: software development managers at large aren't experts in software development work now, and they often weren't expert workers in the days when they were hands-on. They were the people who came to dislike the work enough to want to escape to software development management. They usually don't know enough about the details of the work to understand how subtle decisions can result in tremendous outcomes, and which decisions lead to tremendously good outcomes, or tremendously poor outcomes.

Today's software development managers largely haven't made a study of managing software development and the principles of product development work and work management. Software continues to be managed as one would manage material assembly or manufacturing. Our current generation of managers isn't aware of this problem because software development management hasn't really been a field of interest for most of them - it's just the job that they have, and it's one that keeps them from having to work in the painful niggling details of poorly-managed software.

None of this suggests that we need to wipe the halls of software development management clean and start from scratch with newly-installed personnel lifted from the rank-and-file. One of the worst mistakes that our current management ideology continues to make is the failure to recognize the material cost of losing institutional knowledge when we lose any worker (NOTE: the flip side of this problem is the colloquial horror of not knowing who to hire, and bringing people into our organizations that we dismiss in short order).

The software development mangers that we have today may have become burned out on the details of software development, but those details can be changed to make the work less tiresome, and make the work less like an uphill battle being fought on a cliff face rather than a gentle slope. Their institutional knowledge is far too valuable to lose. Their many years of experience - if paired with a sharper understanding of which decisions make software development good and which make it bad - provides a perspective of the whole of the effort that is invaluable.

It's entirely reasonable to expect that we can transform the software development industry. The root of the problem is in the software development management system. When software development managers are willing to look without flinching into the reality of their situation and the circumstances that have led to their positions, then we can begin to rehabilitate the system. When that happens, many of the software development industry's most futile pathologies will fall by the wayside and we'll begin to tap the potential that is this industry's unclaimed birthright.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Video: Ruby for .NET Developers (From the Norwegian Developers Conference)

2010-06-24T08:30:00.002-05:00

My presentation at the Norwegian Developers Conference, "Ruby for .NET Developers" is available for viewing on Vimeo.

There's a bit of off-color humor in the talk for the sake of making a point about orthodoxy, and done specifically-so for the European audience that it was presented to. So, no offense intended! :)

View it here or open it on Vimeo's site for more viewing options.

Here's the description of the presentation:

After having spent many years coding in C#, and after having spent equally as much time in the C# language culture, Ruby seemed like a lot of bad ideas and heresy. In fact, much of Ruby is heretical to a C# or VB.NET mono–culture, but the productivity gains demonstrated by Ruby on Rails teams remains an unavoidable elephant in the room. This presentation looks at C# code examples side by side with some equivalent Ruby code and shines a little light on what it means to have either "ceremony" and "essence". It challenges the claims of static typing's effect on tooling to deliver "developer productivity". And finally, some examples of Ruby meta programming are given to demonstrate direct solutions to programming problems that would require much ado with restrictions in C# that don't end up doing much more than reducing the efficiency of software development efforts.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Productive by Design

2010-02-10T13:49:00.004-06:00

(Continued from: How the Mainstream Lost Software Development Productivity)

Productive designs are compartmentalized. The parts of your software are shaped to the way that your mind holds on to them. "Abstraction" means the same thing to software as to thinking. "Concern" is another word the shows how software reflects thinking. A software "concern" is something that I have put my focus on that is my present concern.

The compartmentalized bits of software are the shapes of your software's parts. The shape of your software and its parts is only one aspect of your software's design. The shapes of your software's parts is a "static design". "Schema" is another word for it. The other aspect of your software's design has to do with what happens when we bring the parts together, give them a problem to solve, and give them the green light.

We lose software development productivity to relearning and to misunderstanding. The design quality issues that makes it hard to learn, re-learn, and understand your software also make it hard to understand how it works and whether it works. Powerful tools that optimize the moment of creation of the parts of your software enhance your ability to lower productivity.

"Module" is another word for "compartment". A class is a module. Modular design has to be productive not only in its shape, which helps us understand how the software is broken down into concepts, but also in how those parts do some work, and how they work together, and whether they leave their tools out where you can trip on them in the dark.

Software is a machine. It's not a machine like your car's engine or your cordless drill, or like any machine that is made with material. Analogies to physical machines break down almost immediately. There are no machines in physical space that are anything like software machines. They're machines that run individuals' personal way of thinking of how to solve a problem. There are as many ways to program a solution as there are programmers.

Software is a machine. It's made of parts. The word "soft" in software says that it's easy to not only change the working of the parts, but also the shapes of the parts, and to even move the workings to a different part (which often then tells you to change the shape again).

The first great matter of software development is proving that the new shape of some part of your software will still do what you expect it to do when you turn the machine on. The second great matter of software development is whether the shape that you've given to the software will continue to make sense, and that the workings of the software are placed in the right parts.

There's one gigantic difference between working on software machines and working on physical machines that bears mention:

The parts of physical machines wear out, and we replace them with another, new copy of the same part. This doesn't happen with the software parts that you make. When we change software parts, we don't replace ones that have worn out with new parts from the same mold. When we change software software parts, we change the very shape of those parts, often re-shaping the parts around them as well, and moving the workings of one part to a different part, or even a new part created on the spot as part of a new compartmentalization. In relation to a physical machine, software is much more like the mold than the product. Software development is closer to die making than stamping.

Every time you make a change to a software machine, there's a good chance that the change shows up in an adjacent part, and there's a good chance that it'll show up when you're not looking at it. It's a lot easier to get the code you're looking at right than the code your not looking at. That's what "control and observe" is talking about.

To increase the chances of software working as expected after you've tried to turn a part of it into a hat, a broach, or a pterodactyl, you make observations of it. If the only way that you can observe the functioning of the part is to bring the whole machine online, your productivity is many times less than what it should be. If you can make easy observations of the part that you're working on (and maybe parts in the vicinity), you can also make those observations while you're working on it so that not all of the steps are wild leaps into the yonder.

Imagine software that routes baggage from an airline check-in desk through the bowels of an airport's network of conveyor belts and scanners until it arrives at the right gate for the airplane that you're traveling on. Let's say you reshape the part of the software that figures out whether to route baggage to gate 1 in terminal A or to gate 2 in terminal A. Same terminal, different gate. You're losing productivity if you have to start the process from the part of the software that decides to route the baggage between terminal A or terminal B.

Here's the problem:

You need to observe the gate router, but you must control both the gate router and the terminal router. Controlling the terminal router now is wasted work.

You need to feed the terminal router with the circumstances so that you're controlling it to choose the terminal A route. You also feed it the circumstances that it passes on to the gate router so that it does the thing that you expect it to, and so that you can observe that. The terminal router is superfluous in this scenario. You're concerned with the gate router, but you also have to be concerned with the terminal router's workings, even to the point of being concerned with how it gathers the information from its inner workings to pass to the gate router.

If your concern is the workings of the gate router, then you should only have to control the gate router, feeding it the information it needs directly without going through any intermediaries. It's what "separation of concerns" is talking about. Collusion of concerns makes you write more code (and more complex code) to make observations about your software, but it also throws attention switching into work that is trying to be a feedback loop.

If you have a gate routing module, you can likely control it and observe it directly. But what happens when you want to control and observe the terminal routing module? The terminal router will immediately pass the baggage off to the gate router after it has done its work. Can you control the workings of the terminal router without also having to feed it the right information that it needs to feed to the gate router so that the gate router doesn't accidentally raise an error? In other words, how do you control and observe the terminal router without having to be concerned about what the gate router needs to know to do it's job. If there was an accident rate in software development like there is in other production jobs, it would be measured by how often we accidentally have our attention sucked too far into a wood chipper of dependency inversion.

With the right modularity, you've addressed the "static design". The "dynamic design" is how the parts of the software are put together to make the machine. A software machine is put together at runtime, usually by something called a front controller. A Main method is an example of a front controller. So is the thing that receives an HTTP request and sends its instructions it to the right part of your application. This is the part of design that a lot of developers don't recognize as the sore spot of their productivity problems.

There some esoteric terms for this issue of controlling which modules you have to be concerned with when making observations. Much of the language is more complicated than the actual work that you have to do. If you don't get distracted by the lingo, you'll find that this is not only one of the most important problems to solve, but also one of the easiest!

If the part of your software machine that routes baggage from the check-in desk to the right terminal passes control to the part of the software that routes the baggage to the right gate, then the terminal router will likely need to call a method on the gate router (or send a message to it). If I just want to control and observe the terminal router without having to be concerned with the gate router, then I've got problem that can be solved by getting the terminal router to use a dummy gate router instead of the real gate router. I can't change the terminal router code so that it doesn't use a gate router at all when I'm trying to make these observations, because that's not the code that would run live anyway. It wouldn't be a real observation of the thing that I'm concerned with in the first place.

If the terminal router uses a dummy gate router, I don't have to be nearly as concerned with it as I am with my primary concern, which is the terminal router. This is what "separation of concerns" is talking about.

The last thing that you have to solve is a way to switch the gate routing part for a dummy gate router. That's not a hard problem to solve. If you have a terminal router object, you can create a constructor that accepts a fake gate router. The terminal router's default constructor can create a real gate router. When you need to observe the terminal router's workings, create a dummy gate router object that doesn't have to be more talented than not raising errors when it's not supposed to and pass it to the terminal router's constructor.

The term for this pattern is called "inversion of control", or "dependency inversion", or "don't distract me with superfluous concerns".

This is a really simple way to get superfluous concerns out of the way so that they don't interfere with the parts of your software that you're actually concerned with. The above technique is sometimes called "poor man's inversion of control". There are many free libraries that you can use to automate all this stuff and make describing how the parts fit together effortless (or better, unnecessary). They are "dependency injection frameworks" or "inversion of control containers". I like the word "composer".

The dynamic side of productive software design is all about putting the parts of the machine together when your system starts up (or on-demand). The effort that you put into the static parts of the design - the shapes, the geometry, the abstractions - is paid off when you're in complete control of which parts are in-motion when you need to make observations of your software. And to belabor the point: it's the ability to control and observe your software that will give you more productivity, or to restore your productivity. You can't use what you can't understand, and you can't understand what you can't control and observe.

If you can't directly control and observe your software, you might be mistaking efficiency for productivity. You're optimizing your work within a range of improvement that is too close to the floor that make a difference in how you to get the lid off the cookie jar.

Productivity is right there for you to reach out and harvest. It's locked up in your code like potential energy waiting to be turned into kinetic energy when you free it from its moorings. You can find it in all the ways that design interferes with your ability to work with the software that you already have. There's little value in optimizing the pace that you create new software if you're optimizing the pace that you incur compound interest.

Productivity is usually something that you free from your software rather than something you add to software. You create the interference. If you don't want the interference, don't add it to your software.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

How the Mainstream Lost Software Development Productivity

2010-02-08T04:54:00.006-06:00

(Continued from: Denying Productivity)

The greatest software development productivity comes from software that can be easily and readily understood. When you need to make changes to software, you can easily and readily understand the repercussions of those changes. When you need to add new abilities to software, you can easily and readily understand where the additions need to be made. When you're new to an existing project, you can easily understand the code that has already been written by your new team. Understanding how to design software so that you're able to easily control and observe it gives you the ability to create software that can be understood, and software that doesn't obstruct productivity.

Despite the number of software developers who understand the mechanics and principles of software design that cultivate software development productivity, the vast majority of software developers don't understand how to wield the techniques, or aren't convinced by the promises, or simply haven't heard about higher order developer productivity at all, let alone how to get it.

Somewhere along the line, the software development mainstream lost its access to productivity, and the vacuum was filed with counter productivity or productivity proxies.

Two things contributed to the mass extinction of software development productivity in the mainstream: Eggheads and Shills. The eggheads allowed a divide to open between the intellectual haves and the have-nots, and the shills exploited that gap, filling it with misinformation and misrepresentations, verily redefining "productivity" and supplanting it with mere efficiency.

Yes, these labels are chosen for shock value. Specifically because this is an issue that is largely rooted in unconsciousness: the unconsciousness of failing to communicate, and the unconsciousness of deeper meaning.

I've learned quite a bit about software development and design from some very insightful and intelligent people. I'm grateful to the authors who've been my long distance mentors and who have provided me with inspiration. I've met many of them in person over the past half decade or so and these encounters have deepened my pursuit of understanding ever further.

Now that I have a much better understanding of what it is that they taught me, I'm shocked at how poorly this material was often communicated and at how our self-indulgent vanity continues to get in the way of broader communication of ideas that are vital to contemporary society's productivity at a time when productivity is so desperately needed.

I use the term "Egghead" with some affection and admiration, but I also use it in the hopes that some of the great teachers of software development and design will come to learn that they've barely even scratched the surface of the full audience that they need to reach. Eggheads write and speak for other eggheads. The language they chose and the names given to design principles, as well as practices and patterns are stimulating to other eggheads and yet utterly obstructive to the other part of the software development population. When you measure the Egghead population against the mainstream population, the Egghead population doesn't appear to be much more than a rounding error. There's a lot of work to do, and we can do it much better.

Eggheads preach to the choir, and this is often so stimulating that they never leave the echo chamber. They don't learn how to reach the mainstream. They hope that the people they do reach will somehow reach the mainstream, but those people usually end up emulating their egghead heros and become eggheads themselves. The circle expands slightly, but not merely enough to have the meaningful effect on software development productivity that their knowledge and understanding should already have had.

Their communication style is rife with self-stimulating, overly-academic legalese that does little more than enforce an over-estimation of the value of the terminology itself rather than subjugate the terminology to the absolute must of communicating simple and powerful ideas to the mainstream. Eggheads don't like to engage directly with the mainstream. They don't learn to talk to the mainstream and how to teach the mainstream. In this, their self-indulgence leads to terrific loss for human society and is quite possibly one of the most negligent acts perpetrated against the potential for software development productivity.

The divide between the Egghead echo chamber and the mainstream has been readily filled by Shills. While the Eggheads fascinate themselves with themselves, the Shills convince the mainstream to buy snake oil solutions to problems that could be easily solved with plain old soap and water.

The potential for abuse and the leadership vacuum left behind by the Eggheads' self-glorification is readily taken advantage of by Shills willing to convince the mainstream that mere efficiency tools are good solutions for productivity problems. They further exasperate the productivity problems that result from applying efficiency solutions to productivity problems and use the ensuing mainstream customer panic to justify yet more efficiency tools. The mainstream becomes ever more isolated from more meaningful understandings of software development productivity and continues to spin out of control on the back of software systems and software projects that are themselves out of control.

Between the Eggheads and the Shills, things look pretty grim. Those who have good answers to the software crisis are unwilling to forgo the stimulating academic formalism in favor of connecting with the mainstream, and those who would take advantage of ignorance do take advantage of ignorance.

Neither of these two archetypes can be counted on to fix the problem. Eggheads are far too ensconced in the comforts of elitism and Shills have built behemoth networks of companies all selling the same pack of lies, damned lies, and demos. The momentum and justifications of either aren't likely to change anytime soon.

What's needed is a new generation of productivity missionaries who are willing to master the field of knowledge and are willing to learn to connect with the mainstream. They're willing forgo the all-consuming pursuit of elite celebrity and serve society by taking on the software crisis head-on as a matter of honor and duty in the face of continuing negligence and abuse on all sides. They are willing to speak plainly and openly and forgo the perceived legitimization that comes from falling into step with esoteric formalism.

The mainstream has lost track of software development productivity, but it's not so far removed that it can't be recovered. It just can't be recovered through the continued self-indulgence underpinning both the Egghead and the Shill entitlement to ease and winnings in the face of disastrous effects on modern productivity. It can be recovered if pathfinders and communicators are willing to get dirty amongst the "unwashed masses" for an even greater return, and to stand fast against the encroachments of suboptimal local efficiencies bundled with six-figure price tags and promises of yet more software development pain.

Next: Productive by Design

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Denying Productivity

2010-02-08T01:36:00.011-06:00

(Continued from: Cause and Effect and Developer Productivity)

If you choose to use the ability to control and observe your software to restore and cultivate software development productivity, you have to come to terms with the reality of your software as it presently is, and dispense with the stories you tell yourself about just how much productivity your software development has. It's difficult to accept where you are if where you are is much further back than you had realized. If you don't accept where you are, you won't progress. If you already feel that you've arrived, you won't screw up the courage to challenge what it is that you know, and embark on that next stage of your career voyage.

In the past, we've used the term "testability" to describe the quality of design that allows for the kinds of expectations for software development productivity caused by the ability to control and observe. To those who introduced the term to every day software development vernacular, the term has a very specific meaning, and refers to designs that are very specifically evidenced by a set of well-known and observable design principles and patterns. And to others, the term was misinterpreted and misrepresented as software that is tested.

"Testability" means software that is extremely easy to test rather that software that merely has tests. Any software can be tested, what matters is how incredibly easy it is to do so. The easier it is to test software, the more that the software can be controlled and observed. The more directly that bits of your software can be controlled and observed, the more directly that your software can be understood, and the more that your software design will be simple and clear, and reflect principles productive software design.

The level of simplicity in question is usually beyond the realm of possibility for developers who have not had experience with software that is designed to be quickly and readily understood. When the ability to understand software through the ability to control and observe is understood to be the root cause of software development productivity, the software designs that you create will be radically different than what you may have become accustomed to. And while these designs will seem foreign and strange at first, you'll soon learn to see them as the most natural, workable designs that you've created, and that you've worked with.

For example, there's no doubt that you can control the content of an application's database from the application's user interface, but that is a far cry from a level of control that will have any significant effect on productivity, not to mention that such a means to control and observe doesn't even begin to guarantee the extent to which all the bits of software in between can be immediately understood. It's still control, and it's still control that can followed up by observation, but it isn't direct control and direct observation. If you're concerned with the behavior of your data access logic, for example, you must be able to control it directly by using your data access logic directly rather than using it transitively through an application's user interface. If you must use your application's user interface to observe the results of the control you exert on your application's database, then you're not able to satisfy your concern directly through direct observation, and you're invariably not gaining any of the great productivity benefits that come from productive design.

Be careful of getting ahead of yourself: it doesn't lead to learning of any meaningful significance. Learning to control and observe will bring the biggest advances to your experience of software development productivity. If you stake a claim to this learning before you've barely begun, chances are you'll never really learn what it means and how to use it to any significant degree. And this would be sad, because the means to direct control over your software really isn't difficult to learn. You will be disrupted more by the mass of unlearning of what you already hold firmly as software productivity than by the small amount new learning that you'll really need.

If you want to reach into this realm of productivity that is beyond what you've yet experienced, then commit yourself in earnest to be a student of software design principles. Learn how to apply simple software design principles and exercises to productivity. Commit to a journey to master software design from a principled perspective where productivity is your a-priory concern and the thing that you delivery consistently. You'll find that there's really not a lot to learn once you're over the initial interference of what you presently believe.

It's vital that when you embark on the exploration of the next realm of software development productivity that you start from a realistic place, and resist every urge to claim victory before you've barely out of sight of your own home port. If you stake a claim to understanding what it is to control and observe your software, you'll deny the vast wealth of productivity that is practically within your reach.

It's hard to start down a new path to new understanding, but denying that you need to continue learning and to challenge entrenched orthodoxy is no way to start.

Next: How the Mainstream Lost Software Development Productivity

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Cause and Effect and Developer Productivity

2010-02-07T21:47:00.005-06:00

(Continued from: Mistaking Efficiency for Productivity)

By mistaking developer efficiency for developer productivity, we end up creating less productivity. It's an easy trap to fall into. The very shape of your organization is an amalgamation of past responses to productivity problems. Efforts to increase productivity that don't penetrate to the depths of the problem but only reach so far as efficiencies localized to particular job functions not only fail to solve productivity problems, they also fail to fix the counter-productive organizational problems that are the unfortunate side effects of previous naive attempts at treating productivity problems to local efficiency solutions. The organizational problems are also contributors to the vicious circle that worsens software development productivity.

Trying to solve productivity problems by chasing efficiencies that are tied to a particular job function and are not coordinated with other job functions is like swimming in quicksand: even though your right hand and your left hand are doing everything right for everything you know about swimming, the swimming motions applied to the actual medium that you're in are just as likely to make you sink. Swimming is the wrong response to being immersed in quicksand. No matter what your panicing mind is telling you to do, it's just grasping at desperate straws of well-established habits rather than assessing this new situation with responses that might be new to you, and thus quite likely less informed by existing habit. Applying the wrong physics to a medium is almost guaranteed to create side effects that defeat the progress you make against the productivity problem you're trying to solve.

The software development productivity problems that you face are due to a productivity balance sheet that fails to tally the losses as well as the gains. If you simply add more gains to the balance sheet without any regard to the losses, you're not going to understand why the bottom line keeps getting smaller while you're continuing to invest ever more energy into the individual parts of your software development effort. It's not enough to get a bigger bucket to bail the water out of a sinking ship if you're not paying any attention whether the that hole letting the water in is bigger than your biggest bucket. Most software productivity efforts end up snatching a fire axe off the wall and desperately trying to hack another hole in the hull to let the rising water out.

If your organization chases local efficiency solutions to productivity problems, it will end up fracturing itself along job functions that are not entirely separate job functions, but rather individual items on job completion checklists for cohesive workgroups or single workers. As your organization adds to the weight of the processes that petition the productivity spirits to unleash more of this mysterious magical substance from whence we know not, it invariably creates more high ceremony work for managers, eventually calcifying them into clerical functionary roles rather than freeing them to be effective and experienced teachers. The increased ceremonial workload leaves your organization devoid of the leaders who ensure that your organization cultivates successive generations of master workers.

The increased management of ceremony makes it seem as if these managers are doing so much work that the organization around the workload deserves to be promoted to its own department. The resulting departmentalization creates even more of the handoffs and coordination problems that are the fuel for the productivity bonfire consuming your project's allowance of money, people, and motivation.

Your organization will tend to become a fractured collection of organizational shards as it spends more and more of its energy dealing with the mounting productivity losses rather than on producing. This kind of organization produces more internal processes than goods and services that it can exchange with customers for money. Worse, it dries up the pool of knowledge, insight, and vision that creates compelling products and attractive, compelling organizations that compete successfully for more good people and that transform their people into the next generation of success makers.

If you're working off of a one-sided balance sheet, you're failing to connect cause and effect. Failing to connect cause and effect is one of the most common behaviors of contemporary software development when it comes to productivity.

The one-sided balance sheet is a glaring genetic marker of software development that has not been able to connect the effects of it's efforts to the causes of impoverished productivity. The more that one-sided, uncoordinated efforts for productivity fracture organizations along unnatural lines, the more difficult it becomes for software development workers to see the causes of their effects, and the more that dramatic under-performance becomes remarkable. This kind of organization has forgotten the vast wealth of productivity that is its birthright.

The real productivity that you should be experiencing is far beyond the level of productivity machinations that are typically entertained today. The greatest drawback to the promise of software development productivity is just how far-fetched the proposition of potential productivity is. It's multiples greater or orders of magnitude greater than what has become common today.

Amidst all the unbelievable claims of missed productivity is an anchor that can hold you fast to reality of high performance software development: the answers to productivity are incredibly simple, incredibly straight-forward, and incredibly easy to put into practice. There's no magic, there's no mysticism. There are just plain old software design principles at the heart of the matter. Principles that, while seemingly named in a fury of self-indulgent academic terminology fetishism, are nonetheless very easy to understand in practice when they are connected to cause and effect in software development productivity. They are simple principles that can be brought into effect in software through even simpler exercises. Deceptively simple exercises, in fact.

But first, cause and effect have to be connected so that the actions and habits that create the detrimental effects are recognized, understood, and curtailed, and the actions and habits that create productivity are practiced, understood, reinforced, and communicated to others.

First things first: the people who create software must be required to prove that the software that they create can quickly and easily be proven to meet the expectations of it. Expectations for just how quickly and easily these proofs can be made will seem daunting at first, as the organizational fracturing that has been allowed to infect software development as a result of one-sided productivity balance sheets has cultivated a generation of software development that has lost touch with the design simplicity that allows software creators to assume the burden of proof of their own work.

The moment that developers are reconnected with the burden of proof of their own work, software development productivity restoration begins in earnest. To restore software development productivity to your organization, you must reclaim the responsibility of the burden of proof for your work. You do this by learning the dead-simple techniques that cultivate the ability to control and observe your software.

In order to provide a proof that your software works, you need to be able to be able to rapidly control and observe your software. This will invariably change the designs that you tend to use, and guide you toward more simple and clear designs that end up being more easily understood by others, which even further improves software development productivity. Software that isn't easily-understood is the single most important subject in software development productivity. The inability to easily and readily understand software at a glance is a huge part of the productivity loss that goes unaccounted in software development management.

Because of the exasperated fracturing of your organization into departments that are not natural reflections of productive software development, and that contribute to the productivity losses on your software development performance balance sheet, you will not have developed a sensitivity to productive designs. That sensitivity will come. It will come with practice. And while you're practicing control and observe techniques, you're breaking down the barriers that keep both your software and your organization from its productive potential.

The moment you assume the burden of proof, you take steps to radically re-shape your software and your organization to a level of sustained productivity that you would not have presumed possible. By learning to control and observe your software ever more effectively you'll naturally close the gap between the cause of productivity and it's effects.

Next: Denying Productivity

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Mistaking Efficiency for Productivity

2010-02-04T02:24:00.012-06:00

(Continued from: Controlled Productivity)

When productivity starts to decrease and costs start to increase, we tend to panic. In that panic we often accidentally take measures that increase developer efficiency. In the process, we usually end up creating obstructions that decrease productivity.

Productivity that comes from the ability to control and observe is rooted in software design. Specifically, designs that make controlling and observing simple, clear, and very easy.

When productivity starts to fail, you will invariably find software that is difficult to control and observe. You can restore and sustain productivity by exercising controlling and observing. The very act of making your software more controllable will displace less productive designs with more productive designs.

Without connecting the effect of software development productivity to the cause of controlling and observing, you're likely to end up chasing efficiency rather than productivity, and exacerbating the productivity problems you are trying to solve. Most of what we believe about developer productivity has little effect on the ability for developers to control and observe their software. Most efforts for developer productivity end up increasing the pace at which developers create software that is beyond control.

Developer Efficiency

There's a downside to focusing on and increasing the efficiency of code production: it creates a discoordination between the production of code and everything else around it, like testing, operations, planning, and design. If you can increase the pace of not only coding, but also increase the pace of everything after it and before it at the same rate, productivity increases without risking the detrimental side effects of localized efficiency.

Inevitably, increasing the pace of coding without increasing the pace of the whole of software production creates in-process inventory. In-process inventory has even more deleterious effects: It creates rework. The time spent in rework is lost productivity. You're not producing new value when you're busy reworking stuff that is already supposed to be creating value. Decreased productivity creates more panic, and more panic often leads to more ill-fated efforts to rectify the problem by increasing developer efficiency.

If your wheels are spinning in mud, pressing harder on the accelerator doesn't free you from the problem, and it will likely just get you deeper entrenched in the problem. The material ejected from the hole that you're digging collects as in-process inventory. Not only do you not get anywhere, you loose ground.

It's not hard to remain in control of productivity. The trick is to know the difference, to remain calm in the face of panic, and to exercise control and observation until you've got software design under control. And ideally, try not to be distracted by elaborate tools promising productivity if in the end all they offer is a more efficient way to spin your wheels in the mud.

Don't panic. The first decision that comes to your mind when you panic in the face of decreasing productivity can lead to the temptation to displace productivity with efficiency. Don't increase the pace that you're loosing control of your software. Settle in and regain control. You'll restore the productivity you lost and find that there's even more to be had.

Of the developer productivity tools that you have adopted, have they noticeably increased the rate at which your organization makes money as a result? The pace that you can continue turn ideas into money is the definition of productivity that matters. If you focus on the pace that you produce code, you're looking at a part of the picture that is usually too narrow to be meaningfully thought of as productivity.

Next: Cause and Effect and Developer Productivity

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Controlled Productivity

2010-02-03T21:42:00.014-06:00

(Continued from: To Control and Observe - Productive Software Development)

You can work at a level of productivity that redefines your present expectations by learning how to intentionally control and observe your software. Controlling and observing is something that every developer does. Doing it consciously makes productivity intentional - it brings productivity squarely under your control and conditions you to see ways of creating even more of it.

You can't use what you don't understand. It's as simple as that. The easier it is to understand how your software works, the easier it is to understand the repercussions of making needed changes to it. The easier it is to understand how your software works, the quicker you can get to work on it and the quicker you can get done. Inevitably you understand how your software works by observing it. In order to observe your software, you have to be able to control it.

Controlling your software means putting it into a state where you can directly run the chunk of code that you're concerned with without having to prepare adjacent chunks of code. If those adjacent chunks of code aren't your direct concern, then you should minimize their interference with the observations you're trying to make and with the understanding that you're trying to develop.

In the end, all software is controllable. If you can open one of your application's screens, fill in some text boxes, click on some buttons, and end up saving some data in your database, then you're controlling your database through those screens. The issue isn't that software is or is not controllable, but how easy it is to control it. You don't get productivity from the mere fact that software can be controlled. You get productivity from how software can be controlled. Some ways of controlling software are incredibly productive - beyond most people's expectations.

The more direct control you have over your software, the more productivity you'll have. Direct control means that if you're trying to run some bit of code in order to observe it, that you only have to setup that bit of code and interact with that bit of code. If your software is designed for controlled productivity, you can do this at all levels of your system, from function libraries, to classes, to layers in n-tier and distributed applications, to user interfaces, to deployment systems, to runtime monitoring.

We're so used to primitive productivity in software development and primitive productivity is so common and so prevalent that we've lost track of basic principles that are still available to us if we step outside of the limitations of our current view of software development - a view that has taken up residence in software development only within the last ten or so years.

The principles are also unfortunately named. The names almost complete obscure their meaning, making incredibly simple and incredibly powerful ideas accessible to only a few people who have a natural penchant for deciphering software development legalese.

Designing Productivity

Higher order productivity - productivity beyond what you'd expect would be possible - is only created by software design. That's it. No tool, no matter how elaborate, can ever create the level of productivity that controllable software can (unless you're applying that tool to your work after having made your software easily-controllable and readily-observable).

To make software controllable, you simply design it in a way that you can have direct control over any bite-sized chunk of it so that you can make observations of it with an absolute minimum of effort and minimum of distractions by concerns other than the one in your crosshairs. The means to designing controllable software is also deceptively simple. It's based on utterly and trivially-easy exercises that can be done with any software development tool whether it costs thousands of dollars, or whether it's absolutely free.

There's a simple way to test if your software can easily be controlled and observed: Choose one bit of important logic buried in your software that is only executed when some if/then statement (or other conditional branch) flows execution to that bit of code. See how much code you have to write before you can run that bit of code, and print out its results. If you have to control not only the code that you're concerned with, but also adjacent code, classes, systems, database tables, logins, etc, then your code is more difficult than necessary to control and observe.

You loose productivity as well as motivation as your system becomes more difficult to control. There's another level of productivity available to you. That level of productivity can increase your effectiveness by many multiples, and maybe even by orders of magnitude.

Next: Mistaking Efficiency for Productivity

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

To Control and Observe - Productive Software Development

2010-02-03T15:22:00.011-06:00

There isn't a productive software development effort where controlling and observing aren't the substance of productivity. If you break down productivity to it's most fundamental element, you'll find that the understanding of how your software works is the essence of it all. Control and observe your software and you understand how it works. When you understand it, you can change it, work with it, and continue working with it. When it gets harder to make observations of how your software works, it becomes harder to remain motivated to do the right thing. In the extreme, we live out software projects in survival mode, and we saddle someone else with the responsibility of observing whether our software works. It's not the only way. It's the worst way.

The more that you can make observations about your software, and the more that those observations can be made in thought-sized chunks, the more easily it is to understand your software. To make an observation about your software, you have to be able to control that bit of software to do the thing who's side effects you want to observe. The easier it is to do that, the easier it is to understand it, and the more productivity you have. The time that you spend deciphering software that should be readily understandable is mostly wasted time. It's time spent that shouldn't be necessary and that can be eliminated.

Software that is hard to understand is software that is hard to control. Software that is hard to control is hard to understand. A small bit of code that you want to observe should be controlled with as little code as possible. If you can run that small bit of code directly without having to also control adjacent code, then you've got code that is inherently easy to control, and a level of productivity that is inherently greater. We tend to overlook the amount of time that software teams spend controlling and observing, and when things start to become difficult, we tell ourselves that we can't afford to pay attention to it. Ironically, paying attention to it solves the root problem.

Here are a couple of examples of code that is easy to control:
- In-memory objects that can be observed without loading their data from the database
- User interface workflow that can be observed without using the user interface
- Business logic events that can be observed without the auditing software making recordings
- Steps of complex business processes that can be observed without starting the process from scratch
- Secure operations that can be observed without having to authenticate
- Payment logic that can be observed without connecting to a payment processor

If I'm concerned with the functioning of in-memory objects, and if I need to load those objects from a database, then I have to add the database to the things I need to control, and to be concerned with as well. This isn't a natural, defacto quality of software, regardless of how common it is.

We generally accept that the lack of control as a defacto quality of software projects. We overestimate the control that we have, and we underestimate the value of increased control. We don't pay it much attention. We believe that that the limits to the control that we have is just part of the very fabric of software development, that it's unchangeable, and that there's no reason to dwell on it. We may not even have an awareness of it, or of the issue. This is the fatal flaw in software development thinking, because it's the issue of control that explains why software projects rapidly loose productivity and why the extent of that productivity loss is so great.

Maintaining the ability to control and observe is a conscious, intentional effort. If you're not on top of it all the time, it slips away. The traditional productivity loss curve and the traditional cost curve subsequently begin to immediately express themselves. You can tell yourself it's just the way it is, or you can realize the root cause of the problem and the solution and not only regain that lost productivity, but then go on to achieve a level of productivity that you wouldn't have believed possible from your previous perspective.

This issue isn't unknown. In fact it's well-known. The most common mistake in production management is not paying attention to how productivity loss happens and focus instead on increasing the efficiency of the work that is done amidst the accelerating losses in effectiveness. This mistake is the basis for most software production management at this point of industrial software development.

Next: Controlled Productivity

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

QA Missed Something

2010-02-03T02:00:00.018-06:00

When a team is closing in on a release it may still find a flaw so terrible that the release opportunity might be missed. After the initial panic settles down, we go looking for the explanation of why a such a significant problem can remain hidden until so late. Inevitably, QA missed something.

Let me tell you something: there's nothing that QA has ever missed that developers didn't miss first.

Blaming testers for not discovering flaws that they did not create while simultaneously believing that we don't have to check our own work while is it fresh in our own minds is dishonest. There's no defect ever found by a tester that wasn't first designed and implemented by a developer.

Software is a production industry that generally still believes that the people who make things and the people who prove that those things are right should be different people doing two parts of a single job at different times.

Part of a tester's job is to cross check the work that has already been checked by the workers who create the software and who have the best, timeliest knowledge of it. When a tester isn't cross-checking, he's analyzing and doing exploratory work - looking for possible gaps in developers' thinking, understanding, and observation. When a developer's responsibility to deliver repeatably-checked work is passed over to a tester, neither is effective.

It's less of a tester's job to check that a developers' code is right, but that their tests are right, or at least that the result of the development so far is right - which includes tests that have been written before final inspection. That work doesn't start after the code has been written, it's starts before, as they work together to ensure that by the time the product and the tests make it to the final inspection, that all the ducks are already in a row, leaving the tester with the space to continue to question exactly what it means for the software to be right, and how to perform final inspections that hold up their part of the shared responsibility for intentional quality and productivity.

Final inspection is the last place that you want to find game-changing flaws and problems. It's the last place you should find them. It happens, but it should be the exception to how development is done, including testing, and the exception to how you treat final inspection.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Analysis: CodePlex Foundation - The Terms of Mutual Surrender

2009-09-11T18:44:00.025-05:00

Microsoft announced the CodePlex Foundation yesterday. The strategy is quite compelling, and frankly, it points to a watershed moment, a turning point for the Microsoft platform and .NET community, as well as for Microsoft.

The CodePlex Foundation initiative translates resoundingly to one great thing: Opportunity.

History

In April of 2001, Microsoft released Beta 1 of the .NET Framework and Visual Studio .NET. At the same time, Jim Newkirk and a couple of industrious .NET early adopters with a track record in agile development in Java released NUnit, an open source unit testing framework for .NET.

Whether NUnit was the first open source application in the .NET ecosystem, who knows? Nonetheless, NUnit was the first open source .NET software for many .NET developers who looked to open source for answers even before .NET itself was released. Jim is presently a CodePlex Foundation advisor and Product Unit Manager for Microsoft's CodePlex open source repository.

To say that the rest is history is a predictable cliche. Whatever has happened in open source in the Microsoft space up until now, as astonishing as many .NET open source accomplishments are, it may pale in the face of the opportunities that are available in light of the CodePlex Foundation initiative.

Of the time between April of 2001 and the present, the following can be said of open source in the Microsoft space: It was a constant battle. Open source was stigmatized. We had to fight to use open source software in our jobs. And when we weren't fighting to use open source, we were fighting to not have to use commercial clones of open source software that often weren't nearly as robust as the open source.

Certainly things have been getting better in the past few years. More organizations have opened up to open source - especially when open source solutions are the best-of-breed - but Microsoft's own lingering reticence toward open source remained the elephant in the room, and a significant influence over the customer community's perception of open source.

And even though Microsoft has been making some inroads into open source over the past few years, Microsoft's own open source has remained constrained. With a couple of exceptions, Microsoft did not accept contributions to its open source from contributors outside of Microsoft. And Microsoft developers were discouraged from or disallowed reviewing open source code for fear of infecting Microsoft products with potential "viral" effects of some open source licenses.

Imitation is the Sincerest Form

In the past eight years, Microsoft has learned a lot from open source software. A number of Microsoft product ideas were proven first by open source projects. However, because Microsoft itself has not been open to open source solutions, and because a good bit of Microsoft's customer community follows Microsoft's lead on many fronts, the only way that Microsoft could provide its customers with the value available to open source users was to develop new products from scratch.

Microsoft's efforts at duplicating the value offered by open source were fraught with problems. Microsoft employees can't look at the open source code, so Microsoft's competing offerings weren't often too terribly competitive. They would be adopted by Microsoft customers nonetheless on the basis of Microsoft's merit as the largest commercial software foundry.

All this leads to an oft-repeated heart-breaking scenario for .NET developers who are already well-versed in best-of-breed .NET solutions that have matured in the open source world: showing up for work one day to learn that the company you work for has chosen to go with an immature, inferior solution that Microsoft had been left with little commercial choice but to build. This sinking feeling is followed up quickly with the realization that a company the size of Microsoft can't release innovations and fixes as fast as an open source project. The Microsoft clones of open source projects progress slowly while the open source solutions drive ahead with innovative and often more productive solutions.

Ultimately, this cycle repeated year after year and often created an ever-deepening insularism in Microsoft that exasperated the problem even further.

Opportunity

The CodePlex Foundation will bring influential open source projects under its auspices. The details aren't clear yet, but it's reasonable to assume that the foundation will support its projects the way that other software foundations support their projects, with protection for these projects as they are used in corporate and commercial contexts and who knows, maybe even some financial support will be part of the deal.

The single greatest opportunity that the CodePlex Foundation represents is an end to orthodox resistance to open source by Microsoft, and its customer community by extension, and the dawn of a new day of .NET where open source can be openly embraced.

In a potential tomorrow, the best tools for the job aren't sources for intellectual property suspicion, talented software craftspeople have greater freedom to use the tools that they have built significant mastery of, methodologies and techniques aren't driven by tool limitations, and innovation is free to move it its own pace, and can do so as a cooperation between industry and community.

And most importantly, in a world where Microsoft begins to embrace open source, it begins to subject itself to the open competitive forces that will make its products better, and make Microsoft itself a leaner, more agile company.

In this world, .NET open source champions aren't relegated to a relatively small backwoods, but are granted the broad regard and respect that is common place in the Java, Ruby, Python, and PHP worlds, among others with rich open source culture and history.

Also consider that in this world, we're one-step closer to the level of comfort with open source for Microsoft where community contributions to Microsoft's MSPL projects can be possible. The arrival of the CodePlex Foundation doesn't provide for this, but it does provide for the next few significant steps on the way.

This effort will bring Microsoft staff into close encounters with open source software and open source software projects, breaking down the barriers that the community has descried for a long, frustrating time.

I understand that in concert with this effort, Microsoft is also changing its policy for open source contributions for its staff, allowing staff to make limited contributions to open source without the requirement of oversight from Microsoft legal staff! This is a significant shift in policy for Microsoft and points to some significant reconsideration of policies and philosophies of the past.

What Price Freedom?

The source of Microsoft's trepidation over open source hasn't changed. The central issue is still, and will remain for the foreseeable future, intellectual property risk. The source of the trepidation for many of Microsoft's customers remains intellectual property risk.

The solution to the problem is as obvious as it is genius, but the price of freedom isn't without some non-trivial compromise, and a challenging new paradigm for open source leaders to consider.

The intellectual property risks can be greatly (if not entirely) mitigated when intellectual property is assigned to an intermediary, and that, among other services, is what the CodePlex Foundation is for.

Put frankly and directly, the CodePlex Foundation is given ownership of the code.

Per the CodePlex Foundation Assignment Agreement:

"Assignor assigns to Foundation its entire right, title, and interest in any copyright rights that attach to the Code and any documentation delivered with the Code."

Per the terms of the agreement, the Foundation grants an irrevocable license to the code to the original owner. Again, from the CodePlex Foundation Assignment Agreement:

"Foundation grants Assignor and its affiliates a perpetual, worldwide, non-exclusive, royalty free, irrevocable license, to reproduce, modify, create derivative works of, display, publicly perform, sublicense and distribute the Code (and derivative works thereof) as Assignor or its affiliates see fit, including the right for Assignor and its affiliates to sublicense the foregoing rights to third parties."

Ultimately, this means that it's business as usual for the open source projects, with the additions of the protections of the CodePlex Foundation, and greater opportunity for project adoption in spaces that were not previously open, and the chance to participate in what might be a more cohesive and cooperative open source community at large in the Microsoft space.

The Inevitable Distrust

There's no avoiding the issues of distrust that will surface from this. The usual suspicion is inevitable considering the history of Microsoft and open source, the invitation to open source projects to give ownership of their code to the CodePlex Foundation, the preponderance of Microsoft staff on the foundation's interim Board of Directors and Board of Advisors, and the branding of the foundation with an existing Microsoft brand: CodePlex (also the name of Microsoft's public open source repository and community site).

Microsoft has a habit of springing things on the community. But then, so does Apple, and so does Google. Love it or hate it, this is how product companies launch products and how they protect themselves. It doesn't always work out so well, and for my money, Microsoft is the least best of these companies when it comes to operating without early and continual customer feedback in product development, but, well, there you have it. That said, the CodePlex Foundation isn't exactly fully operational yet, and for all intents and purposes, this is the opportunity that the CodePlex Foundation has provided for community input, and the foundation staff has gone out of its way to make this point clear on the foundation's website.

The easiest way to staff up the foundation is to do what Microsoft did. It tapped the people who worked to make the CodePlex Foundation happen for interim positions in the leadership of the foundation. It also assigned some serious business acumen to the interim Board of Directors. Having Microsoft staff in-play on the Board of Directors is just a smart, immediately-sustainable thing for Microsoft to have done with the investment that has been set in motion. The Microsoft staff on the Board of Advisors include many people who are friendly to open source and who are personal friends of many people in the open source community.

And there are a few key names that should likely be on that list. Ayende Rahien and Jeremy Miller immediately come to mind - people who have brought a number of influential open source projects to life, and help to bring those projects into enterprises and ISV's around the world. But this thing is just getting started. It's a "soft launch", as one of the members of the Board of Advisors put it.

So why would Microsoft collude the namespace by naming this fledgling foundation after its CodePlex program? Sure, they're both about open source, and sure there's a healthy dose of Microsoft in the mix, but why not make the effort to disambiguate right off the bat?

John Peterson, a veteran software developer, former Microsoft MVP award winner, and attorney, has been helping me to understand the CodePlex Foundation agreements, and I like his take on the CodePlex naming, and why it's a good idea.

Microsoft has donated one million dollars to the foundation to get it up and running. It has an accountability to its stockholders for what it does with its cash. And while one million dollars might not seem to be much compared to Microsoft's bank balance, it's not exactly a trivial amount when it comes to charitable contributions to what appears to be a fairly radical cause.

The perpetuation of the CodePlex brand is just a good investment, and it likely helps this kind of move go down easier with folks in Microsoft's stockholder community who might not entirely understand what this open source kerfuffle is all about. Microsoft is making this thing happen. It's reasonable that it gets to pick the name, and to use a name that highlights other aspects of its open source efforts.

The Windup

I'm not usually the first person to extend unquestioning trust to Microsoft, and I started my day yesterday with the inevitable distrust and backlash, but I think there's something much more significant here that deserves more than just the usual distrust, and might likely be better served with an unusual trust.

Imagine for a moment if this effort had happened five years ago. Think of all of the often-frustrating pieces of the Microsoft stack that we've had to deal with - software influenced by open source, but missing the target because of Microsoft's policies on staff exposure to intellectual property risks - real or imagined.

Think of all the tools and all the libraries that have shipped from Microsoft over the past five years where your response was a despondent, "Oh no, not again." Part of the reason why we have to contend with these tools is that Microsoft, up until now, has not been able to truly learn from the many mature systems from the open source world that it is called to address with offerings of its own that cary Microsoft's implicit protection from intellectual property risks.

Microsoft isn't going to simply change its well-healed habits on a dime, but we're at a moment where turning in the right direction will set in motion the chain of events that will ultimately change Microsoft's culture in regards to it's ability to see, to leverage, and to participate in some of the incredible work that is being done outside of Microsoft's walls.

The Pitch

I'm hoping that the influential open source folks in the .NET community will consider the CodePlex Foundation's invitation, as odd as it may seem, to consider the possibilities for a future where the .NET community at large has the same common sense perspective on open source as the Java community, the Ruby community, and all of the other communities who's no-nonsense perspective on open source we often covet.

Sure there are lots of details to be ironed out with the CodePlex Foundation program, and the next few weeks will be telling in that regard, but it's with the participation of the open source community that the changes that we've been talking about for years can finally get underway.

I don't own an open source project that I've invested years of my life into, but I can guess what it must feel like to have it suggested that ownership of such a project be handed over to a foundation, and a foundation with very close ties to Microsoft at that. The open source community is being asked to meet the resistance half-way, and to hammer out a program that history will recognize as the turning point for Microsoft, it's customers, and almost every aspect of Microsoft community, culture, and product development.

If you're an open source project owner, think of the possibilities of having your framework or your product begin to reshape the expectations for craftsmanship of Microsoft staff and the greater Microsoft community at large. No one is in a position to require an open source leader to assign their copyright to an intermediary, but the first few influential open source leaders who do meet the CodePlex Foundation halfway will set in motion the kinds of pervasive, positive changes that will change all of our lives and careers for the better.

It's a relationship that starts with one hell of a compromise, but it could be the beginning of a beautiful relationship. Possibly even a historic one.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Do Agilists Understand Lean?

2009-07-19T18:53:00.006-05:00

One of my previous bosses, Steven "Doc" List used to tell me that I'm at my best when I'm teaching. Sometimes, Doc's compliment would strike a raw nerve. Programmers on our team were entitled to choose whether on not to learn. This wasn't an explicit policy, but was engrained in our culture.

We valued learned people, but learning and teaching was not part of our culture or our organizational mechanics. We valued ready-made learning when it walked through the door, but our organizational learning didn't go further than the Agile Retrospectives materials and practice that was fashionable at the time. That's no fault of Agile Retrospectives per se. It's a fault of turning it into something fashionable, and inevitably conferring the unconscious orthodoxy onto it that was steadily growing in Agile methodology culture, often obstructing the line of site to the need for a higher order of teaching, learning, and management culture.

Unless team members understand that there is a requirement to be students in an organization, and to study under a teacher, pride and prejudice will likely obstruct the acceptance of a formal student/teacher relationship, and attempts at teaching will very likely devolve into the predictable butting of alpha geek heads over design and process ideas. And this portends obstructions to meaningful and methodical continuous improvement driven by program goals, and a rise of wild, uncontrolled experimentation.

In his book "Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead", John Shook tells the story of a manager who's supervisor once told him, "If the learner hasn't learned, then the teacher hasn't taught".

I used to tell Doc that it is painfully frustrating to teach knowing that our staff understood that they were not required either by culture or by supporting policy to be in the role of learner - especially when program and project expectations were not being met.

It's true that if the learner hasn't learned, then the teacher hasn't taught. It's also true that if the learner doesn't show up, then the teaching doesn't even begin.

Leaders at successful Lean organizations have pointed out that companies who have failed to duplicate Lean successes often do so by trying to adopt Lean as a process improvement effort rather than an effort to create a learning organization. Despite all of the interesting and beneficial mechanical aspects of Lean, Lean is about creating learning organizations. Student/teacher relationships and protocols are a part of learning organizations, with each role having the responsibilities of that role.

There has been a bit of a dust-up in the agile community lately as to whether Kanban work management is necessarily non-agile or anti-agile. The premise being that Kanban can foster environments of directed work, limiting workers' ability to self-organize, and fostering a disrespectful environment for workers as compared to forms of work management and organization common to agile methodologies.

The essential issue here is the issue of respect, but I find that the Agile perspective is willing to take advantage of incomplete and opportunistic definitions of respect. Kanban proponents have rightly pointed out that respect for people is an explicit Lean principle, and that self-organized work is still happening within Kanban workflows.

I see two perspectives talking past each other, and unfortunately, I see Kanban practitioners being backed into a corner, retreating, apologizing for Kanban, and softening the message to make it more palatable by the dominant methodology culture. Ironically, this was the exact position that the Agile was in at the start of the decade when Agile was struggling to make headway against prejudice and misrepresentation by the preceding traditional methodology culture.

Agilists are concerned about returning to the bad old days when disconnected managers directed work from outside the context of doing that work. It's a serious issue that deserves serious concern. It's a serious enough issue that it demands rigor on the part of the mainstream Agile community to engage the effort to understand Lean more deeply than the often cursory glances and biases projected at Lean and Kanban.

I sympathize entirely with the concern of returning to the bad old days of pre-Agile bureaucracy, but I'm equally concerned about the same tendency for Agile bureaucracy to occlude the meaning of Lean and Kanban.

My study of Agile began in 2000 when a mentor from Bell SIGMA turned me on to XP. My day-to-day immersion began in early 2001. I don't want to return to the bad old days either, but I don't want to go forward into a revised, 21st-century kind of bad old days issuing from the same mechanisms of bias and presumption that Agile itself faced, and often continues to face.

I wonder if agilists at large, in the spirit of inspection and adaption, are taking the time to understand Lean and the organizational and cultural context where Kanban thrives. With Agile, we asked organizations and cultures to consider change. I wonder now if Agile is able to respond to the same challenge.

On the surface, the following statement should rile a mainstream agilist. At least, it certainly used to rile me. It riled me enough not to act upon it even when my instincts told me that it might make a world of difference between immanent failure and rescuing a project, its team, and a considerable investment.

It's perfectly acceptable for a manager to direct the team from a position of traditional, hierarchical, directive authority.

I'm taking egregious advantage here in setting this stage. I'm purposefully leaving out the implicit context inherent in Lean. If I looked at the preceding statement through Agile's lens, I might very likely be worried about it. Looking at the statement through Lean's lens, I'm perfectly comfortable with it.

If we don't look at Kanban through Lean's lens, we're committing the anthropological cardinal sin in failing to realize that we're projecting cultural bias on what we're observing. We're even failing to recognize that Lean may have culture and behavior that is based on different assumptions and biases.

Kanban isn't a return to the bad old days of disconnected, directive authority because the position of management in-situ in a Lean organization isn't the same position of management that Agile efforts are commonly called to contend with and the behavior of management that many of Agile's protocols are shaped to deal with.

A manager of a Lean software development team isn't a remote figure who is no longer in the game. The manager is on the team, and he's one of the most competent technologists on the team. The manager in a Lean organization is also a teacher.

A team that includes a manager with directive authority is still self-organizing. The manager is internal to the team. A manager's expectations aren't disconnected from the reality of the work. And when those expectations aren't being met, he can chose to use directive authority to guide the team to counter-measures through teaching. It's also the manager's duty to help people on the team to develop critical thinking skills and instincts that serve problem recognition and resolution in support of the goal.

The expectation for team members to fulfill their duty as students is part of the managers directive authority. Refusal to engage in the protocols of the learning organization is deeply disrespectful to the organization of people as a whole, and to the manager as a person.

Looking at Lean through Agile's lens is perfectly reasonable. It gives a comparative perspective that can help us understand differences and find meaning. But ultimately, Lean should be seen through Lean's lens and should be assessed from the perspective of its native context.

There is a greater issue of respect and disrespect that is inherent in Lean as seen through the Lean lens. Respect is a two-way street. There is the respect for workers and the respect for managers, or teachers. The mutuality of respect is what makes respect possible. When respect ceases to be mutual, it ceases to be sustainable, and will soon disintegrate.

One of the most intractable issues we have in software development cultures is the lack of line management that remains technically-competent. In fact, line managers in software development are often people who choose to escape to management when they discover that they don't like making software.

We're presently dealing with the effects of several generations of software managers who don't really have much of an idea of what software development work is in detail, which means that these managers can't be effective teachers. Workers don't end up with the teaching that makes them effective and makes the work rewarding. The cycle perpetuates itself.

Agile has been a powerful palliative in dealing with this organizational and cultural snag. By putting a firewall between the deleterious effects of directive authority that is too far from the work and the work itself, agile succeeds in restarting the failing heartbeat of getting software made.

In the software industry, Lean is seen as a kind of specialization of Agile, and that's a unique thing for Lean in industry in general. It's also possibly a detriment and maybe even a disadvantage.

Agile is increasingly encumbered by its own presumptions of organization and culture. Agile's biases are slowly fading into background consciousness, becoming unconscious. As Lean is inevitably and unconsciously seen through the lens of colloquial Agile, many of the organizational assumptions and biases of Agile are projected onto Lean without even realizing that these biases are there.

Lean in its essence is a path to critical thinking, but not a solo path. It's a directed path. A Lean organization is a learning organization. It has teachers, students, curricula, and protocols all focused on meeting the expectations that support the organization's holistic goals for productivity and producing.

Lean is a means to find the unasked question. There appears to be an unasked question in the Agile perspective of Kanban and respect: Do we expect that a Lean organization is the same as an Agile organization?

This isn't meant to be a condemnation of Agile, but it is meant to point out that if Agile isn't careful, it will become the same kind of problem that it sought to solve.

On my project with Doc, I was valued as a teacher. I was the person who got executive support for the project. I shared product design responsibilities with our product owner. I would go to bat for the team when hard decisions needed to be advocated to the executive. I did the technical screening of candidates and made my recommendations to the team about hiring. And I was responsible for setting program goals and expectations for technical implementation.

In the end, I parted ways with the team when it became clear to me that the team had become intractably self-determining, which is a potential risk in self-organizing teams when technical competence and directive authority are not invested in the same person. Ultimately, the team failed, a non-trivial percentage of our small company revenue invested in the project was written off, and the entire team was dismissed.

This isn't a typical Agile scenario, but it is very much a possibility faced by teams in situations similar to ours. The failure likely points less to an implicit weakness in Agile and more to an explicit strength in Lean: respect is indeed a cornerstone of success, but if respect isn't holistic, it risks introducing the opportunism that can see to the failure of otherwise meaningful software development efforts.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Lean Reading List

2009-07-17T03:10:00.011-05:00

I read Mary and Tom Poppendieck's first book on Lean Software Development, "Lean Software Development: An Agile Toolkit" in 2005. It went in one eye and out the other.

I was the Software Development track chair for Austin's InnoTech conference in November of that year. Mary was gracious enough to accept our invitation to come to Austin to keynote the track and to moderate a panel.

I listened to Mary's talk and got a few more clarifying tid bits from it, but mostly, I dismissed it at the time as some form agile sideshow that didn't quite measure up to the specifics that XP brought to the table. A number of us went out for burgers with Mary after the conference and continued the conversation, but I don't think anyone in the group was fundamentally moved by Lean.

I wasn't equipped with the experience to see Lean for what it was. Later, I realized the deep hubris that encumbered my thinking and the vanity that would lead me to expect that I could intuit a subject as vast as Lean from a single book, the way I could with XP and Scrum.

I suffered a serious set-back in 2007. My career's pinnacle dream project was flirting with disaster.

The company I worked for faced an intractable intellectual property constraint that limited our product's market to a small fraction of the whole. In November of 2006, I sold the company on an ambitious plan to solve the problem by building our own platform that we would have unlimited rights to sell. We started exploratory work and envisioning in December. The undertaking had significant executive sponsorship. When our board of directors voted to not fund the project, the CEO bought out the board and gave us the green light. We started official work in earnest in January.

In November of 2007, I sounded the failure warning alarm to my management, and I kept sounding it. In January of 2008, with the situation becoming increasingly intractable, I parted ways with the project and the company. Three months later, after having given the team a chance to pull itself together, the project was canceled and the team was dismissed from the company - from the most junior technologist, up to the executive overseeing the project.

I was frustrated by having felt handcuffed by Agile development orthodoxies that no longer fit the problems of the team, and yet were followed mechanically, and reinforced by management that was captivated by its first exposure to Agile.

I told some of the details of my experiences to a friend. We talked about Agile, in many variants, including Lean Software Development. My friend recommended that I read about the Theory of Constraints and recommended "The Goal: A Process of Ongoing Improvement" by Eliyahu Goldratt and Jeff Cox. In reading The Goal, I recognized the detailed mechanical process that I faced in the failed project. At that time, I hadn't connected what I had begin to learn in The Goal with what I know about Agile Development and the gaps in Agile that I had begun to see after seven years of immersion.

Following The Goal, I wanted to read more into some of Lean's roots. I had largely avoided the Toyota literature up till that point, and I wasn't convinced that I would get much from The Toyota Production System. I knew a number of people in my community who had read The Machine that Changed the World and The Toyota Production System, but I never really got the sense that the reading had connected them with the transformative learning that I had experienced in Agile development.

Before committing to throwing myself into the Toyota literature, I wanted a sample of what I might be getting into. I started with an article. I read "Decoding the DNA of the Toyota Production System" by Steven J. Spear and H. Kent Bowen published in the Harvard Business Review.

In this article, the authors talk about the things that companies seemed to miss when trying to duplicate Toyota's successes using Toyota's methods. I'm a sucker for stories of unasked questions. The rest of my reading about Toyota and Lean in general would be an exploration of the unasked question. The authors' message: Toyota's fundamental nature as a learning organization is often overlooked, with undue attention paid to Toyota's more obvious practices and mechanics.

And this is what drew me in to the Toyota literature. Here I saw the parallels to my failed organization, my own nature as a learner, a seeker, a pathfinder, and a teacher. I recognized the many of the intractable problems that I had observed in the behavior of my failed organization.

On my team, it had become an unspoken entitlement conferred that people were not required to take direction. This was largely exasperated by a management approach that was fixated on social experimentation and which was not capable of guiding the technical execution or product design imperatives of the project. It was an Agile methods laboratory that produced no real acceptable features for a year.

I learned in Spear's and Bowen's article that the organizational structure, mechanics, and protocols that I felt would benefit the team were the basis of Toyota's organization and culture as I was coming to understand them.

The Toyota Way by Jeffrey Liker was my first read into the Toyota literature. I chose this book to specifically continue learning about the Toyota DNA rather than dig into Toyota's specific process mechanics. Understanding that focusing on the process mechanics led to common problems in learning and adopting Toyota's methods, I wanted to hold off on the obvious aspects longer.

I read the Toyota Way with the observations of the Harvard Business Review article fresh in mind, as well as the mind-opening lessons about work management and problem solving from The Goal providing a backdrop.

Cautious of becoming yet another Toyota disciple, I took a turn away from Toyota-specific literature and back toward Lean in general and read "Lean Thinking: Banish Waste and Create Wealth in Your Corporation" by James P. Womack and Daniel T. Jones. This book tells the story of the companies and people taught by Womack and Jones as they traveled around the world after writing The Machine that Changed the World. It reinforced and what I had learned about learning culture and continuous improvement as well as organizational structure and process in the books that preceded it.

While this reading and studying was happening, I was also tweeting about my experiences and studies, and having numerous long conversations on the phone with some of my peers who had themselves started studying the same material. I also revisited my failed project with the previous product owner, who was still with the company, and still succeeding in his own work.

If I had read this material in a vacuum, without any of the constant interaction with my professional network, and the continual revisitation of previous failure, I believe it would have been a much less informative and transformative experience. My circle of friends and network of colleagues continues to inform my learning, and I expect that it will continue to do so.

I turned the reading back to software, choosing to read Mary and Tom Poppendieck's second book, "Implementing Lean Software Development: From Concept to Cash". Reading a Lean software book from my vantage at that time was a very different experience than when I had read the first Lean software book. The subject had now come to life. It was tangible, and it was deeper than what I had presumed previously. From here, my perspective of Lean Software Development takes meaning beyond my perspective of XP and Agile culture and mechanics.

I had organized the ALT.NET Open Space Conference the previous year, and hadn't want to simply fall into the trap of trying to duplicate an original moment. I wanted another theme for the second annual conference. Over the course of many of those conversations with friends and colleagues, we talked about the essential force of the ALT.NET movement as something akin to Lean's Continuous Improvement. The theme of the second conference became Continuous Improvement, with hopes that we would come to understand it more, and maybe understand better whether ALT.NET is indeed a Continuous Improvement culture.

I subsequently read another Toyota book, "Extreme Toyota: Radical Contradictions That Drive Success at the World's Best Manufacturer" by Emi Osono, Norihiko Shimizu, Hirotaka Takeuchi. This book takes on seeming contradictions in Toyota's culture and organization, such as the simultaneity of both a flat organization and a rigid hierarchical organization and the imperatives of a learning culture and continuous improvement that unify the two. The book also spoke about the climate of contradiction that Toyota uses to stimulate creativity and problem solving.

Tom and Mary accepted our invitation to come to the Continuous Improvement Conference in Austin and share their experience with learning organizations, software development, product development, scientific method, and leadership. I can't imagine that Mary and Tom got as much out of the experience as we did, but they had a lasting impact on our community.

Mary and Tom often say that Lean Software Development is informed more by the Toyota Product Development System than the Toyota Production System. Dave Laribee got to the Toyota Product Development book before I did and warned me that it was quite dry but worth reading, and it was. "The Toyota Product Development System: Integrating People, Process And Technology" by James M. Morgan and Jeffery K. Liker spoke a great deal more about requirements, design, planning for work, and creating workspaces, as well as the risk-mitigation processes that Toyota uses. This book also goes into greater detail about the Toyota's people and roles, and fostering "towering technical competence".

The TPDS book also talks about Lean leadership and the Chief Engineer role. It further reinforces the notion of managers and group leaders as having great technical competence, often knowing the work of their staff better than they do, and being paramountly responsible for teaching their staff though the scientific method and Toyota's A3 report technique.

Dave read David Anderson's book, "Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results" while I was reading the TPDS book, and I followed the TPDS book with David Anderson's book. I wish I would have read this book years ago, but again my biases obstructed my perception of the value I would get from it. In this book, David Anderson ties the Theory of Constraints directly to software development and translates pull systems and throughput accounting to the work.

Dave turned me on to Cory Ladas' writing on the Lean Software Engineering blog. I read a few articles, and then went back to the start of the blog and read forward. Much of that writing has been compiled into his book "Scrumban - Essays on Kanban Systems for Lean Software Development". Cory’s writing goes quite deep into pull systems and Kanban for software development. This material and David Anderson’s book offer a profound exploration of applying the mechanical aspects of Lean.

I put into practice what I learned both from my own experience and from studying and study groups on a project with a distributed team in Austin starting in August 2008. My instinct at first was to overlay their existing organization and process with Scrum. Instead, I looked for signs of what was keeping the organization back, established some basic measurements, and did my best to teach what I knew about how to solve these problems. It wasn't trivial work, but it reaffirmed my experience and study of both the cultural and behavioral aspects of Lean, as well as the mechanics, such as Kanban. The experience also continued to reaffirm XP practices as the tactical foundation of both Lean and Agile strategies.

While in Austin, Mary and Tom mentioned the book "Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead" by John Shook, which tells the story of the techniques and processes used in learning culture and learning organization, and goes much deeper into the the teacher/student relationship and responsibilities that are the foundation of Lean.

The book that really brought Lean Software Development together for me wasn't a book that is necessarily about software development, but about the kind of work that software development is: Product Development.

I'd heard Mary and Tom say for years that Lean Software Development is a closer kin to Lean Product Development than Lean Production, or in reference to Toyota, Lean Software Development is informed more by the Toyota Product Development System (TPDS) than the Toyota Production System (TPS).

In Don Reinertsen's, "The Principles of Product Development Flow: Second Generation Lean Product Development", many of the instincts, intuitions, and insights that had been guiding my own work were given a voice, and augmented with Don's experience and perspective. This is a book that is as much a watershed moment in my career as Kent Beck's "Extreme Programming Explained: Embrace Change". It's a book that stitches together many disconnected pieces of learning through a number of years of experience and observation, and then builds a new level on top of this reinforced foundation.

I think that this book deserves to be read much earlier in the list than in the chronology of my own studies. Even if you don't get past the half-way point - it's a clear elucidation on persistent problem in software development management, and the failure to manage software development from a product development perspective rather than a manufacturing perspective, and the failure of software managers to recognize that they don't know the difference.

I've had Atul Gawande's, "The Checklist Manifesto: How to Get Things Right" on my desk for the the past nine months. I haven't opened it yet. It's been on my mind.

On a recent visit to the Toyota plant in San Antonio, I met Mark Graban, who was also there for the tour. I asked him about the book, whether he had read it, and to try to get a feel for where I might put this book in my reading priority. I suggested to Mark that I've got a feeling that it might give me some insight into mistake-proofing, and for enriching the use and usability of my Lean work management app, Floverse.

Mark highly-recommended this book, and so, to the top of my list it returns. I'll update this article once it's done.

And there are yet more books that I've been meaning to get to, many of them in the nascent Lean Startup field, where Lean is applied as a methodology to entrepreneurship and product and business startup. These books will likely get a mention in this article at some point along the way as well.

Context

I wrote this article because people in my network asked me for some recommendations on books on Lean. The more I thought about it, the more I thought that listing some titles and links might not be entirely responsible.

The books that I read and the order in which I read them is inseparable from the context in which I read them. This isn't a canonical list and it shouldn't be treated that way. There are many, many more resources available.

I've benefited tremendously from the choices I've made for study and for engaging community to enliven that study. I wholeheartedly recommend the books I've talked about here, and I would even recommend going through them in the order that I read them. In retrospect, the reading order worked well as an evolutionary thread and helped me reinforce a deeper understanding of subtler aspects of Lean while continuing to layer on ever broader knowledge.

Nonetheless, your mileage may vary, and if your experiences are quite different from those I've laid out here, then you might disregard my own learning adventure and concoct your own.

Either way, foster a community to learn with, and sit at the feet of as many masters who'll tolerate your presence. Anything you learn from a book is just material until you light it up with experience (or reflection) and turn it into knowledge. Learning in isolation rarely has the yield of learning enlivened by experience and community. That's not always the case, but if you have a tendency to hide in a cave, understand that much of Lean is a social practice.

I still haven't read The Machine that Changed the World or The Toyota production System. I may read them at some point, but my goal isn't to consume every Toyota book that I can find. My goal is to synthesize as much understanding as I can, and the past two years have been very rewarding in this regard.

I recently founded the Lean Software Austin group, and I'm looking forward to continuing the study and the work in Lean principles and software development as this community grows as a learning organization itself. The story doesn't end here, but the narrative reading list does (for now).

For your convenience, here is an actual list of the reading I referenced in this article:

Lean Software Development, "Lean Software Development: An Agile Toolkit by Mary and Tom Poppendieck
The Goal: A Process of Ongoing Improvement by Eliyahu Goldratt and Jeff Cox
Decoding the DNA of the Toyota Production System by Steven J. Spear and H. Kent Bowen (Harvard Business Review)
The Toyota Way by Jeffrey Liker
Lean Thinking: Banish Waste and Create Wealth in Your Corporation by James P. Womack and Daniel T. Jones
Implementing Lean Software Development: From Concept to Cash by Mary and Tom Poppendieck
Extreme Toyota: Radical Contradictions That Drive Success at the World's Best Manufacturer by Emi Osono, Norihiko Shimizu, Hirotaka Takeuchi
The Toyota Product Development System: Integrating People, Process And Technology by James M. Morgan and Jeffery K. Liker
Agile Management for Software Engineering: Applying the Theory of Constraints for Business Results by David Anderson
Lean Software Engineering by Cory Ladas
Scrumban - Essays on Kanban Systems for Lean Software Development by Cory Ladas
Managing to Learn: Using the A3 Management Process to Solve Problems, Gain Agreement, Mentor, and Lead by John Shook
The Principles of Product Development Flow: Second Generation Lean Product Development by Don Reinertsen
The Checklist Manifesto: How to Get Things Right by Atul Gawande

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

The Myth of Developer Productivity

2009-07-12T00:43:00.002-05:00

There are a couple of predictable approaches to an imminent car crash. You could throw your hands in the air, scream, and hope for the best, or you could keep your hands right where they are and try to pilot your way through it to the last responsible moment. If you're the pilot of a software team, offloading the responsibility for productivity is like taking your hands off the wheel before you've left your driveway.

The quickest way to shut the door to productivity is to try to solve it exclusively as a tools and automation problem with tools that promise "Developer Productivity". Tools and automation are essential parts of getting software done, but unless you have a firm grip on why you have productivity problems, taking someone else's word for why a tool or automation will solve your problems amounts to little more than a shot in the dark. And since my word is inevitably "someone else's word", please look more deeply into the issue beyond this article.

Productivity that Matters

There's productivity that matters and productivity that doesn't matter. That probably seems nonsensical - after all, if you could gain productivity, wouldn't it matter? It depends on wether the productivity you gain causes a commensurate obstruction at some other point in your software development bucket brigade.

One heck of a lot of tools that are sold specifically as "Developer Productivity" tools create productivity increases for the developers in your development pipeline, and kill productivity for downstream work centers, like testing, packaging, shipping, installation, configuration, and operations.

Focusing on developer productivity without considering the effects of developer productivity efforts on the rest of the whole pipeline will usually create momentary productivity; productivity that only developers will feel, and often only for a short period of unsustainable time.

Software teams and organizations continue to fail to realize sustainable productivity by continuing to make improvements in one area of the pipeline without realizing the cause-and-effect relationship between the localized improvements and resulting degradations in other parts of the pipeline.

Sustainable productivity is the only productivity that matters, and it's the only productivity that can withstand ever more continuous improvements.

Productivity is about product. It's the activity quality of product, or producing, or production - the "ivity" of product. It's about producing and the production of the only thing that really matters - the final product that can be used to achieve the business's goals.

If the goal of software development is just to create developer artifacts without ever delivering them, then developer productivity itself in isolation would matter a whole lot more. The problem with "developer productivity" is that it is inherently productivity in isolation. Productivity in isolation is often naive local optima.

Local Optima

Local optima problems happen all the time in software organizations. They happen when we try to make improvements in one area of the whole software development workflow without understanding the effects on other areas of the workflow.

Developer productivity as we know it colloquially is inherently a local optima concern. My first concern is the productivity of the entire fire brigade. The moment that there's some obstruction anywhere in the whole workflow, the problem will spread outward and poison adjacent work centers, rippling outward, and sometimes making quantum leaps into parts of the pipeline that are, on the surface, seemingly disconnected.

It's not enough to just understand that a change in one area of team's workflow will have effects on another area, it's also critical to understand which kinds of effects will create obstructions for the whole effort.

Greater productivity often come less from increasing the speed that we can do the work itself, and more from recognizing and decreasing the obstructions to production and producing. One of the most obvious ways to attack the productivity problem is to reduce rework and the things that lead to it.

Reducing rework means undertaking organizational and cultural changes. It means that the making of software and the proving that it's right can't be allowed to work at dramatically different paces, which is usually the case with most software teams (even teams using Agile methods).

Every time that testers get backed up in their work, untested software piles up in front of testing. That pile of untested software is the unproven foundation that developers will continue to build upon wether or not it’s sound. As the pile grows, the likelihood of rework grows geometrically, proportionate to the size of the pile, and the near certainty that structural design is insufficiently precise grows along with it.

When the making of software and the testing are disconnected from each other, and software makers and software testers work at full speed regardless of whether they're building up piles of risky inventory, then productivity is going to degrade.

In this kind of situation, doing things that increase programmers' speed isn't going to help productivity at all. In fact, the right thing to do is to either slow the programmers down until the testing inventory is cleared, or to have the developers change hats and work with testing to clear the obstruction. Increasing a developer's speed will only exasperate the problem.

Optimized developer productivity without simultaneously optimizing the entire pipeline is local optima.

Developer Productivity Myths

There are a number of tools and libraries that are sold under the "Developer Productivity" banner. These tools actually deliver developer efficiency rather than developer productivity.

Unless a tool's productivity proposition takes local optima into account, it flirts with negligence. If it does so knowingly and willfully, then it flirts with corruption; verily stealing value from its users and customers.

Consider Microsoft's re-interpretations of "Rapid Application Development" that drive its developer tools design:

Microsoft's Visual Studio enables developers to create input forms and data processing visually with little expense of time and effort. Developers get this work done very quickly, but the resulting systems features are inordinately difficult to test. Microsoft also packages tools that are specifically geared for "testers" in a separate product package, perpetuating the conclusion that doing work and proving that the work is right is the jobs of different people, creating the handoff boundaries that invite unproven work to collect as piles of costly inventory.

While this allows developers to be very efficient in the work of creating the code and artifacts that go into a feature, that efficiency isn’t realized as productivity. The software designs created by developer efficiency tools are unnecessarily and excessively difficult to test. This discourages developers from preventing the defects that pile up in front of the testing work center which also steals productivity from testers, exacerbating the problem of basing today's work on last week's unproven decisions.

Software organizations who are seeking and realizing higher productivity - that is, they produce end product to business sustainably and timely - come to understand that the inventories built up around handoffs and segregated work centers must be decreased and ultimately eliminated in order to really reclaim lost productivity. This has a profound effect on the shape of software teams and organizations, and in retrospect we come to see that our beliefs about organizational mechanics and culture were what kept us trapped in superstitious dark ages of software development productivity.

Regardless of whether tools like Microsoft Visual Studio (among others) seem impressive on the surface, failure to assess the effects of these tools on an entire pipeline will inevitably lead to mere momentary efficiencies and local optima. Worse yet, they will obstruct the learning that does in fact lead to productivity that matters, and will pull managers away from valuable work like teaching and facilitating and force them to become inventory managers and expediters.

It's not just the younger teams using Microsoft's visual tools who are subject to this problem either. Even advanced "agile" teams indulge in developer efficiency local optima like the hypercoding enabled by a higher order of developers tools. In some cases, hypercoding has even motivated urgent and wide-ranging changes to frameworks to optimize for a tool's use even in the face of compelling evidence that more meaningful sources of productivity are likely elsewhere. The compelling productivity of Ruby on Rails programmers despite the unavailability of extensive hypercoding tooling is a good example.

There's no one true answer to whether a tool is going to contribute to productivity. Sometimes just the creature comforts afforded by a tool are significant sources of productivity. But then, not all indulgences in creature comforts can be considered productivity enhancers either.

Ultimately, any of these tools can be used in a balanced, leveled software development workflow. However, until a meaningful understanding and representation of productivity takes root in software development, any team at any level of maturity is going to trade productivity that matters for mere efficiency.

Fixing the Problem

We can affect the things we control. When software projects go awry, we exert control. Any time we exert control without considering the impact on the entire fire brigade of software development workflow, we're going to create efficiencies at the expensive of our ability to produce.

The further away you are from the whole of the team and the workflow, the less likely you are to exert control constructively. As a senior manager, you might prescribe a suite of "Developer Productivity" tools after seeing a compelling presentation from a vendor that is specifically geared to affect your sensibilities from your perspective. If you're a developer, you might convince your team to adopt a "Developer Productivity" tool that demonstrably makes you a more efficient coding machine.

The decisions made by people who are too far from the whole are often a coin toss. There's little telling whether a team will perform better in getting products into the hands of the people who need them, and it’s often difficult to connect the decisions with the ultimate outcomes.

If you're encouraging or enforcing a team organization that disconnects the work and workers from the validation of their work, whether analysis work, design work, construction work, construction inspection work, packaging work, installation work, or operations work, you will inevitably create the conditions that encourage inventory build up and the subsequent obstructions, rework and general degradation of productivity. The shallow pursuit or mere localized efficiencies are more likely to happen when work centers fail to be shaped to productivity goals.

Making any of a number of mistakes that trade local efficiencies for productivity not only degrades productivity, but creates a cycle where the degradation accumulates, leading to the typical software cost curve that is ultimately a reflection of the degrading productivity curve.

Fixing the problem isn't trivial because no single local optimization will have a predictable effect, and a set of disconnected local optimizations degrade productivity even faster and even more unpredictably.

There's no doubt that we need to act on all levels, and ultimately this means decomposing the problem and working on different levels of an organization an at different work centers. But the local things we do to fix the problem have to extend from a holistic understanding of the software development system.

If you want to fix your software development system, find the problems with the system and then understand how the parts contribute to the problem. Evaluate the success of each effort to fix the parts by the effect that it has on the system.

Developer productivity can just as easily be a reality as a myth. Developers are obviously significant contributors to producing software. But an approach to "developer productivity" that isn't also an approach to organizational productivity is often not likely to do more than transfer value from your organization’s treasury to the coffers of a vendor who is more than happy to assume ownership of your precious resources.

We can’t get software done without software tools – this is true - but choose wisely. Your whole team’s productivity is at stake.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com

Relearning: The Productivity Problem that We're Not Supposed To Talk About

2009-07-08T00:42:00.013-05:00

Imagine that you had no memory; that everything you learned had to be re-learned again and again as you did your work. If you worked in software development, you wouldn't have to stretch too far to imagine it. Re-learning is so much a part of the moment by moment work of software development that it's considered normal. In fact, it's the unrecognized backdrop that software development plays out in front of.

Because relearning is not recognized as a problem as software development, it's almost never talked about it. And frankly, it's not a welcome subject in polite programmer society. Nonetheless, relearning accounts for a lot of unnecessary lost productivity and waste in software development. It can account for wide swings in productivity loss on most software teams, affecting the manageability of software projects and a truthful and meaningful assessment of programmers' real abilities as designers, analysts, problem solvers, and real contributors.

I'll offer an observation from my own experience and from the anecdotal evidence of others in my professional circle: I would be comfortable saying that half of the lost productivity on software teams comes from re-learning. It can come in the form of poor testability - the inability to easily provide proofs that expectations are met - and it can come in the form of source code that can't be understood at a glance. Even teams who have solved the testability problem still largely suffer from the lack of usability of their code, and the inevitable mass of relearning that comes of it.

First We Scan, and then We Read

Text in a text editor is interactive media; subject to the same fundamental usability principles that apply to a web page, a desktop app, or even a billboard on the side of the highway.

Before users read the content of interactive media they scan the content. Programmers scan
code before they read it. This is the singular human behavior that programmers consistently fail to recognize. It's the fundamental behavior that when recognized, becomes one of the pillars that code usability efforts can be built upon, and the starting point for recouping losses from relearning.

A singular focus on readability in code is a concern that, while encouraging, still misses the point. It's the same point that programmers missed as a human-computer interaction industry grew from the ashes of our continued failed attempts to create productive user experiences.

Usable code dissolves into understanding at a glance. It doesn't need to be coerced into understanding. I like to call this kind of code "soluble" code for the image it brings to mind of program text readily dissolving into awareness and understanding. Soluble code is likely readable code, but readable code isn't soluble unless it's written to be soluble.

Reading code isn't like reading a good article, where you start at the top and read to the bottom, enjoying the experience and being fulfilled by it. Granted there is beautiful code in the world, but the typical reasons for reading code are not the reasons that we read articles, books, papers, and the like.

In fact, the nature of the media that this article is published in, and the context that this media is typically consumed in, is such that you are more than likely to start jumping around the text with your eyes and with your mouse, looking for the nuggets and pearls while avoiding having to consume the entire thing linearly as it is written.

The first thing we do with code is ascertain whether it is indeed the code that we need to be working with in order to accomplish whatever task we've taken on. This even happens when we're reading code for the pleasure of it. The first eyes-on experience with some code usually involves rapid scanning of the code to ascertain if we're in the right place; if
we're at the worksite. Much of this happens pre-cognitively as it does with any media that we've been called to act upon. First we scan to get our bearings, and then we read.

We take high-level structural scans, followed by smaller, more detailed scans, followed by reading. If the code that we're scanning is written so that it only yields its meaning from a detailed read, then not only are we not optimizing for the natural human interaction patterns, we're also not taking advantage of providing the incidental knowledge that can be yielded to someone while they're scanning. This incidental knowledge accumulates and becomes a significant part of the material invested in the metal map of a codebase that a programmer accumulates through exposure to code that yields its meaning at a glance. We retain a codebase's textual geography only when we absorb its meaning.

If we don't code for solubility, we force programmers to take detailed reads in order to tease knowledge and understanding from the text. Forcing these detailed reads doesn't lead to greater advantage down the road. Less meaning is retained when forcing detailed reading into a context where a programmer is instinctively trying to scan.

Scanning is as much about a process of elimination as it is a process of accumulation. Both are happening at once during scanning unless code isn't amenable to these processes. If we fail to take advantage of scanning by failing to write soluble code, we're stealing productivity from our team mates, and from ourselves. On the surface, this seems like a negligible issue, but lack of solubility is responsible for a tremendous amount relearning and degraded productivity.

Solubility as a code style permeates a codebase. It's a pervasive quality; it has a constant, pervasive effect. Small improvements add up to significant advances when they have a pervasive effect.

Making Code Soluble

There are no cookie cutter patterns for making code soluble. Some things are obvious: meaningful symbol names (class names, method names, variable names, etc), and higher-level methods that tell the story of some process that call lower-level methods that contain the details. Both of these go a long way to create soluble code. Soluble code yields its meaning immediately.

The particular refinements that a team will make depends on the team, the product being built, the technology, and a host of other conditions. Pushing further for an canonical definition of soluble code patterns might lead to just as much productivity loss as productivity gain. Further practices, beyond meaningful names and anecdotal methods, are contextual and shouldn't be dropped into the code as if they are interchangeable parts.

Soluble code is an experience like the table of contents in a novel. It offers and allows multiple levels of reading, with each deeper level yielding ever greater detail. A table of contents is an affordance to the reader that acts as a navigational aid, or map, of the text. But a novel isn't program code. A table of contents at the beginning of a novel can be sufficient for that kind of media with its implicit user experiences, whereas program code itself must be it's own table of contents, right down to the very small, five-line, composable methods. The user of a novel is typically well-served by a table of contents whose resolution goes no further than mapping out chapters, expecting the reader to start reading linearly from the beginning of the chapter. But this isn't the case with program code.

When we're scanning, we're mapping the code by the what of the code; what the code does, what it's responsibilities and behaviors are. Soluble code allows a reader to immediately understand what is does before forcing the reader to understand how it does it. Soluble code serves both modes: scanning for what, and reading the how. Code that isn't styled this way largely deprives a user of the process of elimination, the incidental knowledge, and the mapping that can be had from scanning.

One of the worst mistakes that programmers make in writing code is in failing to recognize that more productivity will be spent over the life time of code navigating through the code than will be spent writing the code. It doesn't take much more effort to write soluble code that serves the elimination of relearning. Choosing to not write soluble code means choosing to keep the relearning waste well-entrenched, but there's more to this problem than mere choice.

Resistance to Code Usability

There are reasonable objections to styling code for solubility and usability. The how
of the system ends up spread over many small methods rather than concentrating it in fewer, larger locations. The root cause of the aggravation is often not that the code is factored in
small semantic units, but that the semantic units are not the right ones, or that the factoring is just not good. Yes, this is the you're not doing it right response, and as unfashionable is this response is, it can nonetheless end up being the root cause. Nonetheless, some programmers are just not going to want to get used to soluble code style, and there will be inevitable grumblings from people who prefer to use more traditional, procedural structure. It's not hard to bring usability to code, but it can be discomforting at first - like transitioning to a new programming language.

Programmers aren't traditionally the folks on a software team who have their heads in the usability game. And we've gotten to an unfortunate point in programmer culture where the answer to many subsequent problems with programmer ineffectiveness has been to create further specializations and allow programmers to be responsible for a narrower and narrower set of expectations rather than deal with knowledge problems as organizational and cultural problems.

Expecting programmers to be considerate of code usability creates friction. For many programmers it's going to be as comfortable as thawing out frozen, front-bitten fingers. But it's not just a programmer responsibility. A good chunk of the responsibility for change rests squarely with the surrounding and supporting organization and its protocols and mechanics. There's more to rehabilitating software development productivity than introducing programmers to new coding pattens. Organizations that have a commitment to learning cultures will do much better at this.

The staunchest resistance to efforts to reclaim lost productivity due to relearning will come from hero programmers. Hero programmers are those guys in any organization that can get the job done with any code in any state. They are typically blessed with what seems like a supernaturally high-definition mental map of a codebase. This is their best and worst quality. It's their best quality because they often know where to fix a problem in a codebase and have a reasonable grasp of the myriad side effects that might result. It can be their worst quality because it's often an effect of mild Asperger Syndrome common to programmers, engineers, and jobs that require extended, intense focus. It's often accompanied by the lack of awareness and empathy toward peers typical to the condition.

Hero programmers can suboptimize the efforts of a team by not having to rely on soluble code, often navigating a codebase entirely from memory. The ability can be incredibly useful, but the need to write soluble code rarely manifests because it's not a personal need, and the typical lack of empathy obstructs the ability to recognize how this advantage undermines their team mates' efforts to be as effective.

The code created by hero programmers isn't soluble because the heroes rely on uncommon facilities that preclude the need for solubility. They don't notice the design smells because they rely almost exclusively on echo location for navigation. The resulting work product can often only be as effectively worked on by the heroes themselves, which inevitably leads to predictable resource bottlenecks, bus factor risks, excessive specialization, and the general malaise on a team as the effects of code toxicity spill over into the human realm.

The analogy to the baseball shortstop who was famous for making great plays applies: his coach often pointed out that he was out of position to begin with.

Usability is rooted in an ability to have empathy for users. That empathy gives designers the pause to consider whether they got the user experience right. It's the empathy that leads to the questioning that leads to the recognition of interaction design problems in the form of cognitive obstacles that undermine the user's productivity. In an environment with traditional and institutionalized lack of awareness and lack of empathy, dysfunction can drive unconscionable waste.

Hero programmers rarely stop to question whether they've left behind a good experience for others on the team who need to navigate, then understand, then make changes to the code. And most programmers will fail to recognize the two distinct mindsets that are in play when working with code.

Writer's Mind and Reader's Mind

Without ever doubting whether code is usable, programmers will presume that the right thing has been done. Programmers who are good at creating soluble code have learned to doubt every line of code written. It's not that they have all of the answers. More importantly, they've learned to be constantly questioning.

In the words of the anthropologist Claude Levi-Strauss (not the guy who invented blue jeans), "The scientific mind does not so much provide the the right answers as ask the right questions."

Asking the right questions means constantly switching from the writer's mind to the reader's mind. That means that after each bit of code is written, a programmer switches mental contexts and assesses the code from the perspective of someone who has never seen the code before, asking, "What have I done to undermine the immediacy of someone else's understanding of my work? What unrecognized presumptions have I made about their context as a reader that only applies to my context as a writer?"

That's quite a trick, and it's not uncommon to hear the complaint that it can't be done, but it's what interaction designers do all the time. It's not uncommon for the deleterious effects of programmer autism to cause programmers to fail to have awareness sufficient enough to break out of the laser-like focus on writing code and switch back into questioning.

Code that doesn't incur the cost of relearning is code that can be immediately understood by someone who hasn't seen it before with minimal orientation to the code and the problems it solves. It's code that can be understood at a glance. That code is rarely if ever produced by a mind that has lost its awareness of its context and its mode. It isn't produced by a mind that fails to concede that code is written to be read, that the readers are other humans, and that the reader's context and needs are not the context of the writer - at lest not until the reader has found the worksite in the code, and gained sufficient understanding of the worksite to begin to make the necessary changes that they're tasked with.

The writer's mind is a context that often fails to recognize that the focused and relatively linear mechanics of writing code is quite different than the mechanics of consuming code as a reader. And this is where the misconception over readability comes from. The writer's mind, working relatively linearly is also consuming code in that mode. Readability is a quality that pertains to the linear consumption of code as text, and that kind of optimization of experience is only relevant some of the time in some user scenarios.

Break the Habit

The autistic mind is lulled by the hypnotic cadence of constantly pumping out code. It will complain that the constant switching between the writer's perspective and the reader's perspective is ruining their ability to get in the zone. And in truth, it is, but it's that particular zone that is causing a lot of relearning debt to mount up. There are other zones to get into with equally pleasing effects, but it is a matter of breaking some habits and replacing them with new ones. There are a few tricks and techniques that programmers can use to break out of the fog and get into the zone.

The most important thing to practice is the constant self-questioning of whether the code just created is soluble; that it can be understood at a glance and yields enough meaning while scanning to contribute to the mental map. Instead of presuming that everything I do is made of gold, I presume instead that it's made of fools gold. From that perspective, I can usually gain the right amount of objectivity to assess solubility.

Pair programming and test-driven development are two techniques from Extreme Programming that are extremely effective at clearing the fog. When these techniques are practiced together, it becomes very difficult to be lulled into the unexamined mind that produces unexamined code.

If you don't like pair programming, try using some kind of timer on an interval that is just short enough to be uncomfortable that will remind you to come back up for some critical thinking.

And lastly (for this article anyway; there are more tricks out there), Context Specification is a form of Test-Driven Development that recognizes solubility, usability, reader's mind, and authorship, and forces the issue of contextual analysis to bring more practice of awareness into programming.

Commit to Learning

Ultimately, there's no where left to hide. The anti-productivity that comes from inviting and accepting relearning continues to accumulate. We've pushed it into the lowest level of software development where only programmers can (or may) see it, but it still affects everyone touched by the software project or the product.

Relearning waste is a fundamental organizational behavior. Until we shape organizations around dealing with waste and learning, we continue to fail to see in the range of the spectrum where the sheer magnitude of the relearning waste is visible. Relearning is inculcated into software organizations. Shifting from re-learning organizations to learning organizations and learning culture is how this problem is ultimately solved.

It's self-evident: counter-act relearning with learning. The term learning organization isn't the trite and trivial perspectives that see learning as something external to team and organization; a destination where people are sent once in a while to be "trained". Learning is no more about receiving training than quality assurance is about giving click recorders to test monkeys. The emphasis we put on "training" in the software industry undermines our ability to see the "learning" side of the same issue, and to build truly meaningful teaching/learning experiences in our organizations, and to subordinate organizational mechanics and protocols to these imperatives to counteract the constant, tireless forces that drag us back into relearning.

The value we throw away on relearning is recouped when we counter it with meaningful learning. Learning gets real when it's expressed in every organizational protocol and business process, and importantly for software development, when it is expressed in every single line of code written by everyone involved in bringing a solution to life.

The Problem with Big Design Up Front is the "Big" not the "Up Front"

2009-07-03T22:40:00.001-05:00

The risks of Big Design Up Front isn't the "Up Front" part, it's the "Big" part. Doing too much design without validating it inevitably drives a good bit of the productivity loss that continues to hamper software projects.

It's an issue of large batch sizes - the "Big" in "Big Design Up Front". The larger batch sizes mean that we're always building today's software on yesterday's work and yesterday's decisions before proving that yesterday's work and decisions are sound. The larger the batch size, the larger the risk that the incorrectness of design will lead to ever more expensive countermeasures. And in many cases, design flaws are too subtle to be seen immediately, and collect negative potential energy in the form of on-going degradation in productivity that come to be seen as "normal".

Big Design Up Front has often come to be interpreted as "No Design Up Front", and has led to a lot of mediocrity in design by inappropriately democratizing the responsibility for design quality. This is often done in the name of cross-training, but the best way to teach potential software designers to be good software designers is to constantly expose them to good software design, rather than the mediocre design that can come from a misinterpretation of Big Design Up Front.

Working with software developers and organizations to help realize the potential of software product development through higher productivity, higher quality, and improved customer experience

Learn more about my work and how I can help you at ampgt.com