Accidental Simplicity

Do we care whether it's Software Engineering or Craft?

noreply@blogger.com (Steve Berczuk) — Mon, 23 Jan 2012 11:00:00 +0000

Much like a tenet of agile software development is that "planning is more important than the plan," there are some questions about software development that are useful to explore, even if you can't suggest a good answer. One of these these is whether what we, as software developers do, is (or can be) engineering.

I was talking with a colleague about a comment someone made about a post about lessons that software engineers can learn from artists. In this comment the poster raised an issue that comes up a lot when someone uses the phrase "software engineer." The question was, in essence:

Is what we do when we build software "engineering" or "craft?"

My colleague, who has a background in Naval Engineering and who is engaged to an Electrical Engineer (who designs and builds hardware) suggested that the difference is that one rule of thumb might be:

It's engineering if you need to use calculation (about physical constraints).

An example was that to build a bridge you need to do calculations to determine if the bridge, as designed, will support the load you want to support. Many of the "classical" engineering disciplines (Mechanical, structural, electrical) fit that criterion. But by that definition, some things which most would agree that are crafts, such as carpentry, involve calculations. So there is probably more to it that that.

On the other hand, I don't think that all work that is in a discipline that we might call "engineering" is engineering. A good friend designs test equipment for a living. When he builds a sensor to measure the performance of his home heating system is he doing engineering, or is that activity more like a hobby or craft?

As I think of the question, and the frequency at which it comes up, I wonder:

Why does it matter whether we are software engineers or software craftsmen?

Engineering, to me, implies a certain level of discipline are repeatable process, and all of the classic engineering disciplines are grounded in physical laws that we are fairly confident in for the scales at which they apply.

If the "software engineering" question is important to you, take a minute to understand why you are asking the question, as the reason for asking may affect the answer. You can bring discipline to building something you regardless of whether or not you are engineering. I don't think that the question of how (or whether) to make software into an engineering discipline is an unimportant one. But I do think that it is often asked in a vacuum, much like the an attempt at a user story that is missing the "so that..." clause.

To me, as a professional, it's more important that I bring discipline to what I do. I want to build useful, quality, software where the definition of "quality" that is often most appropriate is Jerry Weinberg's from Quality Software Management: Systems Thinking:

Quality is Value to Some Person.

It might be fair to say that we're not always "crafting" or "engineering" software when we sit down at an IDE and program. Sometimes we're just coding (or hacking, or programming). But if you (and your team) are committed to building software that meets your customers needs, and you are always working to learn how to work better, you're doing a good thing. And you don't need to call it engineering to acknowledge that.

Agility (and Learning Opportunities) Everywhere

noreply@blogger.com (Steve Berczuk) — Sat, 14 Jan 2012 20:00:00 +0000

People often ask "can I apply agile methods to something other than software development?" Since the basic appeal of agile methods is to acknowledge uncertainty by planning in increments, evaluating where you are relative to the plan and other forces, and planning the next increment, it seems like there should be no obstacle to following an agile approach for any project. The lurking question many have is "but can my type of project really be structured in an incremental way?"

While in general, I tend to believe that the answer is "yes," it's always good to have examples. So I was intrigued to head a segment on the radio show Living On Earth, where the rapper Baba Brinkmann described his approach to developing The Rap Guide to Evolution as "Performance, Feedback, Revision," which is a very concise way to describe an agile approach. This meta-aspect of this also intrigued me, as Brinkman used the performance, feedback, revision theme in the evolution rap as well.

This is also another reminder of how software developers can learn much from many seemingly unrelated domains. As Rob Austin and Lee Devin explore in Artful Making: What Managers Need to Know About How Artists Work managers, including software managers, can learn from theatre, and in Computers as Theatre Brenda Laurel discusses what interaction designers can learn from theatre.

So, yes, you can be agile if you are doing something other than software, and if you are developing software you can learn how to build software better by looking at other disciplines.

Questioning before Answering

noreply@blogger.com (Steve Berczuk) — Thu, 08 Dec 2011 16:00:00 +0000

The other day I came across a short video in which a parent is faced with answering a unexpected question posed by her young child. I found this video amusing because, being the parent of a kindergartener, I expect to be faced with many awkward moments like this in the future. I also found it an interesting metaphor for software requirements gathering.

In the video the parent could have answered the question a whole lot more simply has she just asked taken a moment to try to understand what the real question was, and not taken the question at face value. (I'm being purposely vague at this point in case anyone wants to watch the video.)

Most of us have been on projects where much effort has been spent in an attempt to solve a customer's stated problem, only to discover that that either:

The program didn't actually solve any of the customer's real problems.
The team didn't solve the problem that the customer wanted solved.
There was a much simpler solution.

I'm not suggesting that exhaustive requirements capture is the answer. Nor am I suggesting that you should not start building software until you have complete understanding of a problem. The reason that agile methods work well is that they acknowledge uncertainty, and provide a simple path to refining understanding: evaluating working software.

Acknowledging uncertainty does not mean ignoring information. In some cases customers might not really understand what they need, and the best way for a team to progress is to take a guess based on the information they have, and iterate on a working system. In many cases a customer will know the reason they want to do something, but may not be able to step back enough to express it. In these cases, but not trying to ask "why?" your team is ignoring information that can help them do their work. In more cases than not, a customer can fill in the pieces of the user story template:

As a (ROLE) I want to (DO SOMETHING) so that (REASON)

and provide insight into the problem that she wants to solve. This will save time, effort, and help the team deliver a better system.

If your team takes customer requirements statements at face value without exploring the reasons why a customer wants a piece of functionality, they aren't holding up their end of the collaboration part of agile software development.

Had the parent in the video though to ask why the child asked the question that she did, the process of answering might have been simpler.

SCM Tool as Collaboration Tool

noreply@blogger.com (Steve Berczuk) — Mon, 05 Dec 2011 14:00:00 +0000

A few weeks ago I had the opportunity to give a virtual talk at a product launch for an SCM Tool vendor. The theme of the talk was basic branching strategies, and preparing for the talk led me to reflect on what Brad and I wrote in the SCM Patterns Book. This post is based on those thoughts.

When Brad Appleton and I wrote Software Configuration Management Patterns I wanted to focus on how to use Software Configuration Management and Version Control to improve productivity and collaboration. In many organizations SCM processes, procedures, and policies are focused on stakeholders other than those who rely on the system most: developers.

A good SCM process can help you to work better as a team, and help your team be more agile. An ineffective one can slow down development, and be an obstacle to progress. Likewise the tools you use, while secondary to the decisions about the way you work, can affect how effective your SCM process is.

Tools are important. A good SCM tool helps you work more effectively as a team and allows you to focus on your work and not the mechanics of the SCM process. But the process needs to come first to use SCM effectively. Here are some brief thoughts on what I think are the core issues to consider when designing an SCM process.

Code Lines

While there are many aspects to Software Configuration Management, the first concept that comes to mind for many is version control. Developers commit changes to a code line, and update their workspaces from a code line. The main role of an SCM process is to facilitate integration, and you want commits and updates to be frequent to avoid complicated integrations.

Frequent commits also give developers the full benefit of having a history of their work so that it is easy to revert to a known state when a coding experiment goes wrong. So you want processes and policies that encourage frequent integration.

The simplest SCM model is to commit changes into a single code line. Working on a single code line is simple in some ways. But it requires discipline to keep a single code line, with frequent commits, stable.

A code line where people have practices such as tests and a discipline of pre-commit builds in place, is an Active Development Line. Not every team can maintain this discipline. And even for those who can there are legitimate reasons to work on more than one code line. Once developers realized the challenges of working on a single codeline, they quickly become interested in the way a tool supports branching.

Branching

The first question people often ask about a tool is how it supports branching, this question is best answered in the context of understanding why you want to branch. Branching is a technique for enabling parallel development. When you branch you create code line that is a copy of its parent and it can evolve independently. A Branch allows you to work on variation of the code without affecting or being affected by, changes in the parent code line.

The ability to work in parallel can be an effective tool for enhancing productivity. It can also be a cause of bottlenecks and frustration. The difference between the two results depends on whether you follow branching patterns that address the problem that you are trying to address.

Branching can be a very useful mechanism for making your development process more effective if you do it the right ways, with the right tools, and for the right reasons. A poorly applied branching strategy can have the opposite effect, decreasing your velocity and making it harder for team members to work together. So while how a tool supports branching is important, it's more important to understand why you are branching, and if branching is as helpful as you'd like. As I explain in more detail in a article on TechWell:

Branches are helpful if the enable you to work more quickly than a single code line would.

To make this concrete, here are some examples of solutions to a "fix the release" scenario:

Use a Release Line: A common branching pattern is to create a branch when you release software, The Release line allows you to deliver fixes to the released software quickly without having to worry about the impact of new development work on the code. Most of the time a release branch is the right choice for managing changes to a release. There are alternatives, but each adds its own cost.
Defer the fix until the next release is scheduled. This will only be a viable option if the problem isn't critical, or if the team releases software frequently. In many cases, teams don't release software frequently enough to do this.
Fix the problem in the existing code base and release early. This can work only if you have a continuous quality assurance cycle that insures that your code is always ready for release, While many teams strive to do this, few are able to.

So, in many cases, the traditional Release Line pattern is the right choice. But it's good to be aware of alternatives.

Distraction

While branches can be useful, they have costs that you need to be aware of. Team Software development is a balance between integration and isolation. Branching can be a powerful tool for isolating change.

But be aware that not all changes are are isolated as they first appear. For example, much of the time when you fix a problem on released code, you will also want to address the problem in your new development code line. So by working on a branch you have created the need to do the same work twice. And the more branches you have, the greater the risk of distraction, and reduced velocity.

Software configuration management practices and tools are part of a developer's basic toolkit. Like all tools, the right tool used correctly they can help you get your job done. The wrong one used without understanding can distract you from your goal.

Branching can help improve your velocity, but be sure to be aware of the reasons you are branching, and always ask your self if the branch will deliver enough value to justify the cost. and always test and integrate frequently to avoid surprises.

Keep in mind that the real goal of an SCM environment is to help your team to work effectively.

Definitions

noreply@blogger.com (Steve Berczuk) — Tue, 22 Nov 2011 13:30:00 +0000

The other day a colleague use the phrase open interval to refer to time periods that included their endpoints. This didn't sound like the definition that I'd learned in analysis class, which I will admit was not my favorite subject in grad school, so I searched for a definition and found:

An open interval is an interval that does not include its end points.

This led to a brief, but interesting, discussion about why my mathematically-minded colleague flipped the meanings. It came down to the feeling that the definition didn't make sense in terms of common english usage (when you open a door, or a group, for example, you're being inclusive).

Definitions add precision, but when you use words that have meanings in multiple contexts, you can find yourself in confusing conversations. Consider a term like normal, which can be defined as:

a lack of significant deviation from the average

This sounds pretty benign, but people often carry around more that just definitions when they hear words. Using "normal," for example, in conversation can get tricky if the group you are in has not established a baseline context; even though there is no value judgement inherent in the word, people might assign values to being more than a certain distance from the average. And then rather than than discussing the problem at hand you get distracted by a meta conversation.

The same is true when you're adopting new engineering practices. Consider a word like refactoring, which has a precise meaning and that people often use to mean "change." When adopting agile methods, you might have a discussion about how many of the practices that you need to follow to be able to say that you are using a method. Can you do Extreme Programming as a solo programmer when Pair Programming is a practice, for example?

The discussion is very interesting and useful, as it leads to an understanding about what is useful about XP practices. But from a definitional sense, I would err on the side of saying "No, XP is defined as the practices, but you can be doing something XP-inspired." The question you want to ask is "why does it matter if I'm doing XP (or Scrum, or another process)?" Maybe it doesn't. Unless you want to learn the value (and limits) of a practice.

The discussions get more interesting when it comes to concepts like "agile" which are very overloaded in different contexts. You can be more agile from a business perspective, yet not follow a defined agile software development method, for example. But if you want to say that your team is agile, be sure that you understand what your definition is.

Doing an "inspired by" process can be good if you are getting the results you want. When you are learning a process, it's important not to customize the practices until you understand why they are in place. And when communicating with others, being a bit fussy on the definitions (or qualifying your description) can be helpful.

Definitions help us to communicate. It's a useful default to stick with the definition in the domain. Then apply value judgements as needed. It seems better to say that you aren't agile, yet getting better results than you were, than to morph the definition of agile so that it fits your practices. Making "Agile" a checklist item itself is counter to the principles of agile. Define what you do in terms of your practices and evaluate effectiveness in terms of your results.

Words and definitions help with communication. And it's hard to come up with an unambiguous word. But try hard to use words as they are defined in your context. That will help you to communicate, set expectations, and maybe even be more agile.

Refactoring

noreply@blogger.com (Steve Berczuk) — Sun, 20 Nov 2011 15:00:00 +0000

Frank Buschmann wrote a 2-part article in the July/August and September/October issues of IEEE Software that covered an important aspect of software development: refactoring.

Yes, you'll hear the word "refactoring" spoken often in any development team, but most developers rarely discuss what refactoring is, and whether they are refactoring or doing something else, like "reengineering" or even re-writing.

I won't repeat what Frank said, as he said it quite well, but as I read the article I realized that there are some points that get lost in many teams and are worth emphasizing:

Refactoring does not mean "any kind of changing." It means changing the structure of existing code to preserve functionality. In principle you are making the code "better."
To refactor you need a way to validate that you have preserved functionality. Automated tests are probably the best way to do this. I suggest that you can't refactor without having a test plan in place. And from a practical perspective, this means automated tests.
Refactoring is a way to improve code quality, but like other things a development teams spends time on, you you should refactor when it provides business value.

Being precise about when a change is refactoring and when it's reengineering, a rewrite, or simply changing code to add a feature helps set expectations with stakeholders and helps you as a developer better understand when you'll be done. And by thinking in terms of refactoring continuously you can deliver code more quickly, and avoid the need for reengineering or re-writing, and better be able to evaluate functionality with incremental delivery.

Refactoring is good. Not all good coding is refactoring.

Agile or Not: How to Get Things Done

noreply@blogger.com (Steve Berczuk) — Wed, 19 Oct 2011 13:00:00 +0000

Agile software development always felt intuitive to me. Developing software incrementally, in close collaboration with the customer is the obvious way to deal with the uncertainty inherent in both software requirements and implementation. The technical practices of automating necessary but time consuming tests, and deploying, early and often are the obvious ways to give an team the ability to evaluate the functionality you have and to to decide if the software works as expected. And it's also important to decide if what you built still makes sense given the current environment. Agile isn't the only way that people build software, and it may not be a perfect approach, but it's one of the best ways of dealing with a system that has unknowns.

Agile software development acknowledges uncertainty. The ability of agile methods to make it very obvious very quickly when a project, or even a process, is failing makes people uncomfortable. The visibility of the failure leads to a desire to return to more "traditional" processes. For example, a common complaint is that agile organizations don't create "good enough" specifications. Someone might blame an unforeseen problem on the lack of a spec that mentioned it. This is possible, but it's only true if the person writing the spec could have foreseen the problem.

The desire to point back to a lack of specification also points to a lack of buy-in to a fundamental premise of agile: it's a collaboration between the business and the engineering team. Some other possible causes of the problem could be:

The development team didn't test thoroughly enough.
The code was not agile enough, so the bad assumptions were embedded into the code so deeply they were difficult to address.
Communication was bad.

More complete specifications that address all of the issues a system might encounter are one way to build software. But the best that people can do is to write specifications that address what they know. Quite often no one has complete advance knowledge of what a system needs to do.

There are projects where things are known with enough certainty that a waterfall process can work. People can write good specs, the team can implement to those specs, and the end result is as exactly what everyone wanted. I've worked on projects where this was true, and in these cases, the specifications were reviewed and tested as much as code might.

Even projects that seem suited to waterfall fail. For any method to be successful:

Everyone involved needs to be committed to the approach and
There needs to be a feedback loop to correct errors in time.

The second is point is more important than the first, since people will make mistakes. But being committed to the process is what lets people accept and offer constructive feedback. The reason that waterfall projects have such a bad reputation is that many are implemented in a way that results in problems surfacing late.

Agile methods when done well have the advantage of having built in feedback loops, so that customers and teams have ways of identifying problems early. When agile projects fail it's often because people ignore the feedback and let things degrade for longer than necessary. (Failing fast can be considered success!)

So, Agile or not, your process will only work for you if everyone works within the framework to the best of their abilities, and if you have mechanisms in place to help people do the right things. Otherwise you can blame your process, but odds are, that's not where (most of) the fault lies.

More on Being Done

noreply@blogger.com (Steve Berczuk) — Sun, 16 Oct 2011 13:00:00 +0000

Continuing the conversation from last week, Andy Singleton followed up on my post on being done with this post. Which is good as this is one of those questions that sounds simple in theory, but in practice contains some subtlety.

While I was advocating a good definition of "Done" to enable you to measure progress along a path, Andy's point seems to be that many teams don't establish enough of a path. He says:

In my opinion, most agile teams aren't doing "Test Driven Development", and they aren't doing scrum iterations where they plan everything in advance. Instead, they are doing "Release Driven Development." They focus on assembling releases, and they do a lot of planning inside the release cycle.

This is probably true in more cases than not, though one could argue whether if you are not doing iterations or TDD whether you are, in fact, doing agile. But even if can concede (which I'm not sure if I am) that you can do agile without planning and acceptance criteria of some sort, what Andy describes above still sounds like it has an element of plan, execute, adjust, though perhaps in a more chaotic way than a "textbook" scrum process, and and perhaps at a smaller scale. So knowing having a clean definition of what you want to do is still important. In the Indivisible Task I discussed how teams get stuck with planning because it's difficult to think of tasks that are small, discrete, and which produce useful work. It is possible to do so, but it is hard.

Perhaps the disagreement here is more of a misunderstanding. I consider being able to measure progress in terms of completed work items an essential part of gathering the data you need to improve your process and understand how to be more agile. While doing it at a macro (sprint) level is good, doing it on a day-to-day or hour-to-hour basis is essential. If you do this at a fine-grained enough level, and have a good testing infrastructure, you can release software more frequently. So, at the limit, perhaps Andy and I are talking about the same thing.

Since I only summarized, I encourage you to read what Andy has to say for yourself. And I look forward to hearing comments both from Andy and other readers.

Being Done

noreply@blogger.com (Steve Berczuk) — Wed, 12 Oct 2011 13:00:00 +0000

Agile New England (which used to be called the New England Agile Bazaar, and which was started by Ken Schwaber) , has this wonderful activity before the main event each month: they host Agile 101 sessions, where people who know something about agile lead a short (30 minutes) small (about 10 people) class on agile basics for those who want to learn more about some aspect of agile. From time to time I lead a session on Agile Execution, where the goal is to help people understand how to address the following questions:

How can software and other project elements be designed and delivered incrementally? What set of management and technical practices would enable this?
How do you know whether your Agile project will complete on schedule?

When I lead the sessions, I tend to focus on tracking, defining stories in terms of vertical slices and the importance of continuous integration and testing to making your estimates trackable. Since the classes are so small and since the attendees have diverse experiences, the classes are sometimes more of a conversation than a lecture, and I find that I learn a lot, and sometimes find myself rethinking what I know (or at least exploring things that I thought I understood well enough).

During the October 2011 meeting I found myself reconsidering the value of defining "done" when writing User Stories. I always have thought that defining done is essential to tracking progress. But what done means is still a difficult question. Andy Singleton of Assembla suggested that

The only useful definition of done is that you approved it to release, in whatever form

While the goal of agile methods is releasing software, I find that this definition, while appealing in its simplicity, misses some things:

Agile methods have a goal of continuously shippable code. Of course, "shippable" might not mean "ready to release" and cane mean something closer to "runnable," but you can get there by doing no work since the end of the last release. That isn't the goal of agile.
With that large scale definition of "done" you have no way of tracking progress within a sprint.
Without an agreement on what you're going to, it's hard to know when you are changing direction. And acknowledging change is an important part of being agile.

The last point about acknowledging change isn't just about "blame" for things not working out. It's about evaluating how well you understand both the business and technical aspects of your project, and it forms the basic for improvement.

True, having incremental definitions of done that you can use to track progress does help manage budgets. But that really is the least important aspect of having a good definition of done. Even if I were on a project with an infinite time and money budget, I'd want to have a sense of what our goals are.

Having an agreement among all of the stakeholders on what being "done" means lets me:

Improve communication among team members and between team members and business stakeholders.
Evaluate my understanding of the problem and help me identify what I can improve.
Set expectations so that it's easier to develop trust between stakeholders and the engineering team that the team can, and will, deliver.

"Ready for Release" is a key component of "done" and and essential part of being agile. But it's not enough.

See Andy's response, and read more in Part 2 of this conversation.

Continuous Learning, Coaching, and Learning from Others

noreply@blogger.com (Steve Berczuk) — Mon, 10 Oct 2011 13:00:00 +0000

There was an article in the Boston Globe this week by Scott Kirshner: Staying Competitive in the Workplace that emphasized the importance of keeping your skills up to date.

It's a short article and worth a read. Some of the activities Kirshner suggests are similar to those Atul Gawande makes in the appendix of his book Better: A Surgeon's Notes on Performance.

Related to this theme is a New Yorker article by Gawande, Personal Best, on the advantages of challenges of engaging someone to coach you in your profession. I continue to be amazed at how much I'm learning from Gawande, a surgeon, about how to be a better software engineer. I suspect that I first realized this when I started learning about Patterns. (The short post, The Pattern Technology of Christopher Alexander by Michael Mehaffy and Nikos Salingaros discusses how an architect influences the software development community.)

Maybe the common theme of all these writings is that it's important to be ready to learn, and you can learn from the people you least expect to.

SSQ Article on SCM and Tools

noreply@blogger.com (Steve Berczuk) — Sun, 09 Oct 2011 19:00:00 +0000

I recently was interviewed for an article on SCM and Tools that Crystal Bedell wrote for Search Software Quality. Updating tools and processes key to overcoming SCM challenges is brief, and makes some good points about the relative value of tools compared to understanding what you are trying to accomplish with your process.

The best SCM tool, from a day-to-day perspective is the one that is the most invisible to developers, and the best tool really can't help you much if you have a process that just makes it hard to collaborate. This article was also validation that trying to be too agnostic when Brad Appleton and I wrote the SCM Patterns book was a good idea. While some tools make some practices easier, a tool can't replace having an understanding of the reasons to use (or more important, not use) techniques.

Read the article, and if you want to learn some basic SCM practices read the book. For those who want a very comprehensive discussion of branching (which seems to be, for better or worse, the topic that people most struggle with when using SCM tools), Streamed Lines is a good place to star.

Thoughts on The Lean Startup

noreply@blogger.com (Steve Berczuk) — Sun, 18 Sep 2011 23:00:00 +0000

I had the opportunity to get a review copy of The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries. I learned a few things from the book, but the most surprising thing was that what, on the surface is a book targeted at business people, entrepreneurs in particular, actually had a lot to offer developers and project and product managers who work at all sorts of companies. In addition to reviewing many of the lean principles and practices familiar to anyone who takes agile software development seriously, this book also makes a case for how practices that work for developers (like testing) also make a great deal of sense at the business level. To be sure the tests are different, but the idea of having measurable expectations is something all agile developers who have written a unit or integration test should be familiar with. Eric Ries shows how this same approach makes sense for determining the direction a business should take.

This book provides guidance on the tools to use to answer the question "are you building the right product" quickly. Even if you are on a team in an established company -- even if your are building tools for internal use -- concepts like the Minimum Viable Product can be useful to you immediately.

In spite of the book's business oriented focus, it makes clear the importance of good engineering practices to build a sustainable product development environment. This book can benefit those who work in startups and anyone who works on a team that wants to have the impact of a startup.

Why SCM Patterns is in Patterns Format

noreply@blogger.com (Steve Berczuk) — Mon, 05 Sep 2011 19:00:00 +0000

I recently had an opportunity to speak to an undergraduate software engineering class. The professor for that class, Danny Dig has a very interesting practice of asking practitioners to speak to the class over Skype when an appropriate topic presents itself. Danny invited me to join then when they were discussing software configuration management and version control.

During the session Danny asked me why we used pattern form for the material in Software Configuration Management Patterns: Effective Teamwork, Practical Integration was in pattern format. At the time I might have quickly answered that I was interested in patterns and the book was based on patterns we'd written. But aside from being a too flippant, and not terribly profound. that answer isn't actually right.

There are two aspects of patterns that make them well suited for software configuration management practices, especially those at a team level:

Patterns describe things people have done successfully, and we wanted to document the practices that worked well, and which, when they were missing, caused problems. We wanted to catalog the things that we found ourselves explaining again and again in different organizations. If you read the book, much of what you read will sound obvious. But it's only obvious if you know it, and may people don't.
The pattern form makes it very clear than everything fits into a context, which includes other decisions you've made and the environment your working in. People often get captivated by solutions without really thinking about whether the solution is right for he problem they have, and what else you need to do before and after to ensure success.

Too often people associate SCM with tools. and techniques without considering what techniques work well for the team. SCM Patterns puts the solutions in context with your organization, process, and architecture and (hopefully) provides you with some guidance on the steps to build an environment where people can code together effectively.

The other thing I realized shorty after writing the book is that, while the book was in it's final stages just as the Agile Manifesto was being worked on, the book had a lot to say about agile collaboration. The SCM Patterns book, to me, is really about how to work together more effectively. SCM is just the means.

Experience and Learning

noreply@blogger.com (Steve Berczuk) — Mon, 11 Jul 2011 11:00:00 +0000

In the past few months I've heard a couple of stories about (in effect) the disadvantages of experience when it comes to innovation and productivity. A Story on WBUR on July 5, 2011
discussed how venture capitalists tend to favor young entrepreneurs, as having never learned the wrong things in business they don't know what's possible or impossible. In one quote a VC said:

One thing I love about these people is they don’t know what they don’t know. They don’t fear failure. They don’t mind risk...

A March 6, 2011 story on NPR on the pros and cons of raising the retirement age made reference to to an article in Foreign Policy which asserted that younger workers have advantages in the workforce since they learned more recent technology in school.

While new skills and new perspectives can add a lot to a team, is the best way to get these skills to simply hire people who know only what they learned in school? Is anyone you know who is a successful, productive, software developer only working with skills and perspectives that they had when they graduated school? I suspect that the answer is "no." And do people who are new to a field fall often fall into avoidable traps because they don't have the experience to know that the traps were lurking?

In any field it's important to be continually learning. I've worked with recent grads who seems to now be aware of important, new technical subjects, and with people with 30+ years of experience who were the people I learned many new things from.

The important thing in both of these cases isn't "new-ness or experience" but an ability to keep an open mind, learn, and "embrace change" (to borrow an expression from an agile software development method).

In the book The Cat Who Walks through Walls, the subject of the title could walk through walls because she was too young to know that she could not. While having this mindset is useful, there are ways to achieve it without being inexperienced or ignorant.

As the WBUR story concluded: "In this day and age, forget about age. All you need to start is a fresh idea."

To be a successful agile team you need a mix of perspectives and experiences and a willingness to learn from each other.

Happiness and Agility

noreply@blogger.com (Steve Berczuk) — Thu, 07 Jul 2011 11:00:00 +0000

Agile development practices at their core, have a common theme of making better use of the time spent developing software. This starts at the project level and continues down to the developer day-to-day-activities. Consider an agile iteration. The team starts the iteration with a clear sense of the priorities for the sprint, and pretty good understanding of the project scope. Having estimated and committed to getting the work done, the team also has a sense that the goal is attainable. The team members then collaborate to get the work done as a team.

While we can see how this might make for an efficient delivery process, consider how agile practices relate to enjoyment and morale on the team. In the paper If Money Doesn't Make You Happy, Consider Time, Jennifer Aaker, Melanie Rudd, and Cassie Mogilner discuss five principles for spending time in a happiness-maximizing way. Some of these seem like a bit of as stretch, but there is a lot to be said about the relationship between a workplace using agile practices agile, and the teams being effective and happy. Consider how the 5 principles could map to agile practices:

Spend your time with the right people. If you on a collaborative, self organizing team, committed to reaching a common goal, it's hard to argue that you are with the wrong people.
Spend your time on the right activities. The authors of the paper also use the phrase maximizing the value of the present moment. If you've done your planning right, you have a good sense of what you are working on both for the sprint and each day. If your team is working on a project with an involved product owner, then you know that you are working on something useful.
Enjoy the experience without spending the time. The premise here is that one can feel pleasure by thinking of a good result. Activities ranging from planning, to test driven development to retrospectives seem to fit here.
Expand your time. This principle is about having control over your time. Since the team is self organizing and decides how to get work done, you should have a fair amount of control over what you are working on and when.

Using an agile process can't guarantee happiness, or even that you team does all of these things. You can have a poorly defined product backlog, a less than collaborative team environment, or a micromanaging product owner or manage, for example.

But agile is more than just a product planning process that increases predictability It can also increase productivity and (as Brian Marick said during an Agile 2008 Keynote) Joy.

Specialization, Generalization, and Effectiveness in Software Teams: Clinical Metaphors

noreply@blogger.com (Steve Berczuk) — Mon, 04 Jul 2011 14:00:00 +0000

I was thinking about the relative value to a team of a developer with specific skills (say UI development) versus adding someone who was more of an end-to-end developer. Two stories about medical practice that provided some insight into the question.

I recently read Better, Atul Gwande's book about improvement in medical professions. In this book he relates a story of a clinic in rural India that is poorly staffed and funded, and where doctors manage to successfully perform procedures that are outside the realm of their training. They are able to practice these skills effectively and saved lives.

A story on This American Life, discussed a doctor in Kermit, Texas who practices in areas beyond his stated training, with poor results. In the end nurses who worked with the doctor filed complaints against him for mistreating patients. This doctor worked alone, and collected information from questionable sources, and ignored the availability of better qualified resource near by.

In both situations doctors practiced beyond their training, yet in one case this improved care, in the other it degraded care. Two of the key differences seemed to be that the doctors in India frequently met to discuss each other's cases and learn from each other, and the doctors in india worked beyond their area of specialization in order to keep the clinic functioning and performing useful work. There was no way to wait for a specialist to do the work, and still keep the patient alive. Gawande credits the camaraderie of the group in India as well as their daily discussions of their cases, their decisions, and why they made those decisions, for their ability to improve their range of practice.

Even though the dynamics in software teams differ from those in a hospital, there are lessons from these stories that software teams can apply to better develop teams of generalizing specialists, and more effective agile developers:

An ethic of continual learning, both from outside resources and each other can help a team of generalizing specialists to be effective; Having the ability to work across a stack isn't as important as a willingness and ability to learn how to fill gaps in knowledge.
The flow of the project is important to consider when deciding who should work on a task, consider the overall flow of the project. If the specialist is available, there you may as well have the "expert" work on the task (after the team agrees on an approach ). But if waiting for the specialist would delay the project, and there are other people on the team who don't have a full backlog of work, consider having someone else work on it. The specialist can provide advice, and perhaps improve on the work later. But you will have functional software sooner.
It's important to take time to review and discuss decisions as a cross-functional team to understand what you did, and what you can do better next time.

So it seems like it's best add to a person to a team who has a skill that the team might be weak in, but who enjoy working on the whole application when the team thinks it makes sense. Self-organizing, cross-functional teams are a central part of agile practice, and the combination is important. Just having cross functional people doesn't work; they need to work as part of a team working towards a common goal.

Tips, Habits, Customs and Agility

noreply@blogger.com (Steve Berczuk) — Tue, 28 Jun 2011 14:15:00 +0000

I was listening to a Planet Money Podcast about tipping recently. According to the story, the custom of tipping has persisted even though the reasons that the practice was established no longer apply. According to the Planet Money story both waiters and customers think that tipping is useful, evidence to the contrary aside. The discussion led me to think about practices some teams do in the name of following a process.

Habits and routines are useful because they free you to focus on the important tasks. Rituals and processes take a somewhat irrelevant decisions out of your hands, and conventions make it easier for others to understand code and other artifacts. And when you are starting a new approach to work, following the rules by rote can help you understand the method. But circumstances change, and when the reason for the ritual has changed, there are a few possibilities:

The practice is still useful, for reasons you didn't anticipate when you started it.
The practice adds less value, but it doesn't add any overhead so there is no harm in continuing.
The practice leads the team to spend energy addressing a problem that does not exist any more and causes waste.

To avoid instances of the last case that take up too much of you time, it is important to periodically reflect on why you are doing what you are doing, and it it still makes sense in your environment. As your team adopts new tools, new practices, take time to think about whether what you are doing still makes sense. Periodic retrospectives are one way to capture this, so it's important to set aside time to review how you work. (And even to set aside time to review how you do retrospectives!)

Lots of teams (and individual) follow process steps with good intentions, yet they end up causing unnecessary overhead. For example coding conventions that made perfect sense in the days of fixed length variable names, and text editors that add overhead when you have refactoring IDEs and follow clear naming conventions.

The podcast also reminded me of one of of my favorite Gerry Weinberg quotes (which I referenced in Software Configuration Management Patterns):

Survival Rules are not stupid; they are simply over-generalizations or rules we once needed for survival. We don't don't to simply throw them away. Survival rules can be transformed into less powerful forms so that we ca still use their wisdom without becoming incongruent

(from Quality Software Management: First-Order Measurement)

Traditions and habits have their place. But if you are doing something simply out of habit you can benefit from acknowledging that fact.

Better: More Parallels between Medicine and Agile Software

noreply@blogger.com (Steve Berczuk) — Mon, 30 May 2011 18:40:00 +0000

I recently finished reading the book Better: A Surgeon's Notes on Performance by Atul Gawande. Having read The Checklist Manifesto: How to Get Things Right (which I commented on earlier) I expected to learn not just about medical practice, but also about parallels between medicine and software development. Much of what Gawande says about performance in medical environments has can be said to apply to agile software teams, in particular, the need to focus on core practices, the value of retrospectives, and the value of generalists.

A discussion of how hand washing, a simple technique that is vital to infection control but that also requires culture change to implement with the discipline required to be effective, brought to mind how the challenges teams have being agile often center on the challenges of having teams begin to apply basic practices, without customization.

In the discussions of Polio vaccination and malpractice were excellent non-software examples of how we can benefit from retrospective practice, especially the prime directive, which reminds us that to improve we can't focus on blame, but rather on how decisions we made and how to make them better. A discussion of medical practice in war zones provided excellent anecdotes to relate when someone says that they can't gather data to measure performance and thus improve; Gawande relates how combat doctors, those with the best excuses to not have time to do "paperwork" somehow find time to enter the data needed to track and improve the care of soldiers, while gathering data in hospitals seems to be hard.

In a profession where specialization is valued, Gawande discusses time spent in India, where understaffed hospitals mean that surgeons can't afford to specialize and yet get do well by collaboration, cross-training, and discussion; in short, how "generalizing specialists" help to eliminate roadblocks and help get the most value from a team.

Like all metaphors and comparisons, there are gaps. Doctors and Software Developers do different things, and work under different constraints. But by focusing on the differences you miss an opportunity to learn from those around you. This, to me, is the key lesson in Better.

Displaying Build Numbers in Grails Apps

noreply@blogger.com (Steve Berczuk) — Wed, 11 May 2011 23:00:00 +0000

Being a fan of Continuous Delivery, identifiable builds, and Continuous Integration: I like to deploy web apps with a visible build number, or some other way of identifying the version. For example, having the build number on the login screen for example. In the Maven/Java world, this is straightforward. Or at least I know the idioms. I struggled with this a bit while working on a Grails app, and wanted to share my solution. There may be other, better, solutions, but the ones I found approaches that didn't quite work they way that I'd hoped.

My requirements were:

To display a build number from my CI tool, where the number was passed in on the command line. In Bamboo, for example you might configure a grails build as
-Dbuild.number=${bamboo.buildNumber} war
To only change build artifacts and not any source files.
To not misuse the app version, or change the names of any artifacts.
To be simple and idiomatic.

I realized that that Grails itself changes the application metadata (application.properties) when it does a build. So I modeled this approach after code in one the gant scripts in the Grails distribution.
The steps are:

Hook into a build event to add a property in to application.properties.
Access the property using the gsp tag.
Pass the value into the the grails war command using a Java property.

To create the event hook, create a file in your project/scripts directory called _Events.groovy that contains a method like:


eventCreateWarStart = { warName, stagingDir ->
    def buildNumber = System.getProperty("build.number", "CUSTOM")
    println("Setting BUILD_NUMBER to "\+ buildNumber)
    ant.propertyfile(
       file:"${stagingDir}/WEB-INF/classes/application.properties") {
         entry(key:"build.number", value:buildNumber)
   }
}

Then you can access the property with a GSP tag like this



<g:meta name="build.number">

And you can now display build numbers (or any other properties that you want to pass in from you CI system) in your Grails app. This worked for me with Grails 1.3.7. The only down sides are:

This works on a per-project basis (That is probably easily fixable)
The properties that we set are hard coded. (this could be fixed with a convention that iterates through all of the system properties that start with "build" for example).

I welcome any suggestions for improving this, or alerts that I'm using the entirely wrong idiom :)

Using Issue Tracking Systems Well

noreply@blogger.com (Steve Berczuk) — Mon, 14 Mar 2011 23:00:00 +0000

Many of us in the agile community have mixed emotions about issue tracking and issue tracking systems. Tracking is good, but issue tracking systems can become time sinks if not used well. I shared some thoughts on this topic in an article on Sticky Minds.com. Enjoy, and please comment (either here, or on the Sticky Minds site.)

Recent Writings (Dev Ops and Testing)

noreply@blogger.com (Steve Berczuk) — Mon, 28 Feb 2011 23:29:00 +0000

I recently wrote a couple of articles that might be of interest to those who read this Blogs.

On Sticky Minds.com, (a TechWell community) I wrote about unit testing, and how to overcome the inertia that stops people from testing in The Value of Really Dumb Tests

I also wrote an article for CM Crossroads that gives an overview of DevOps.

If you read the articles and find them useful (or not) please comment here at the article site.

Continuous Delivery

noreply@blogger.com (Steve Berczuk) — Thu, 23 Dec 2010 17:30:00 +0000

If you didn't already know that the key to reliably deploy quality software is to take a cross-functional, full-lifecycle approach, Jez Humble and David Farley's book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation will help you to understand. Much like Jim Coplien describes in Lean Architecture: for Agile Software Development the "secret" to successful lean projects is to work with "Everyone, All Together, from Early On."

While the authors have experience in, an a pre-disposition for, agile techniques, the principles described in this book apply to any organization, whatever the process, though if you take the approach to heart, you will find yourself becoming more agile, which is to say, more responsive to customer needs.

The book covers the full lifecycle from requirements, and design, and coding, to acceptance testing, deployment and operations. There are even discussions on test design, data migration and performance optimization and capacity planning. All of this underscores the point that the goal of building software is delivering it to users, a point which some other books understate. Continuous Delivery develops the concept of a Deployment Pipeline, showing you the impact development and testing practices have on deployment and operations, and emphasizing the value of involving the operations team early in the development process.

There is a lot of material here, but there are also pointers to other resources. In the SCM and release management are, the authors build on works including Software Configuration Management Patterns: Effective Teamwork, Practical Integration, Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers) and Configuration Management Best Practices: Practical Methods that Work in the Real World.

Not only do you walk away from this book with an understanding of how to start implementing a continuous deployment pipeline, you may also find yourself writing down a list of things to try, and tools to use. While not a tool-centric book, the authors provide many examples of tools to help you implement each phase of the process.

Having written about the version and release management process before, and how it relates to architecture and organization, I was excited get a review copy and see the authors give a thorough discussion of how the SCM and Release Management practices relate to the rest of the software development ecosystem.

While having a lot of material, the book is well organized and written. It's not a quick read, and you'll want to have a notebook or post-its handy to capture the idea it helps you to generate, but if you are interested in improving your deployment process you will find this book very valuable, whether you are a developer, tester, release engineer, or someone who manages people in those roles. When you finish this book you will not only have knowledge about how to implement a deployment pipeline, but also the encouragement to know that it is possible, no matter how complex your project might seem.

What Agile QA Really Does: Testing Requirements

noreply@blogger.com (Steve Berczuk) — Sun, 28 Nov 2010 23:45:00 +0000

Teams transitioning to agile struggle with understanding the role of QA. Much of what you read about agile focuses developer testing. Every project I've worked on which had good QA people had a much higher velocity, and higher quality. And there are limits to what developer Unit Testing can test.

In an "ideal" agile environment a feature goes from User Story, to Use Case, to implementation. With tests along the way, you should be fairly confident that by the time you mark a story done that it meets the technical requirements you were working from. If someone is testing your application after this point, and the developer tests are good, it seems like they should be testing something other than the code, which has already been tested.

Even though it's important that your QA team does not become the place where "testing is done" as that will sidetrack an agile project very quickly, it is possible that the QA Team is testing (ideally in collaboration with developers) things that the team decided that developers could not test completely. Also, in practice, problems sometimes slip through developer tests, and the QA team is then testing the developer tests to give feedback on the types of things the developer tests need to be better at.

Aside from "catching the errors that slipped through" a QA team doing exploratory testing is also exploring novel paths through the application, and may discover problems during some of these paths. In this case the team and stakeholders need to have a conversation about whether the error is something that users will see and care about, or whether it is something that isn't worth fixing.

In effect, exploratory testing is testing the requirements for completeness.

If you have a QA team, and they are doing exploratory testing, they are really testing:

Requirements: Finding interactions between features and components that were not defined or understood when coding and developer testing began.
Developer Tests: Identifying where developer tests were not as good as they could have been (and how they could be better). This is a good use of automated "integration" tests.
Systems tests that might be hard to test in a developer context.This might be the one set of tests that are the primary domain of QA.

And, as I've said in other places, QA team can be extremely valuable testing user stories and requirements before they make it into a sprint backlog, by providing feedback about testability and precision. The QA team can also add value by testing how useful a product that meets the specification actually is.

Of course, this is an idealization. But if you are disciplined about doing developer testing, you can still get a lot of value from exploratory testing, whether the people doing them are dedicated QA people, or people who are filling the role.

Risks of Manual Integration Testing in the Context Rapid Change

noreply@blogger.com (Steve Berczuk) — Mon, 22 Nov 2010 14:00:00 +0000

You have probably come across a situation like this: It's close to a release deadline. The QA team is testing. Developers as testing and fixing problems, and everyone is focused on getting the best product they can out the door on time. During this time, you may notice that someone on the QA team, while working late has found an interesting problem. And they clearly spent a lot of time investigating the problem, identifying the expected results, and the details of why what's happening is wrong. If this is a data intensive application, there may well be SQL queries included to allow you to pinpoint the issue quickly. In short, an ideal problem report.

Except for one thing. Your team found and fixed the problem hours before. And the effort to find and document the problem could have been spent on something else.

It's hard to avoid this kind of overlap when you don't have a complete end-to end automated testing process. And it's probably impossible (for now, anyway) to create an automated test process that will completely replace exploratory testing. Any manual testing process has to balance the timeliness with the "flow" of the testers. If you update a deployment hourly, then you'll reduce the risk of redundant bug reports, but the testers will experience too many interruptions. If you have a paradigm where you code, deploy, and stop coding until you see the next set of issues, but that means time wasted, and probably reduced quality. The answer is better communication between the the testers and developers about the state of the application. Under ordinary circumstances, you might have this sort of exchange in your daily scrum.

You're still early in adopting agile, you'll likely have a week or two at the end of a project where you'll have more effort put into manual testing, and that manual testing will find errors. So the simple thing to do is to query among the team before spending too much time documenting an issue.

Some of the options are:

Searching the issue tracking system. This sound like a good idea, but sometimes it's hard to find the right query. And it's possible that something got fixed without and issue being generated.
Asking. Yelling out a question, sending and email, or posting a message in a team chat room gives you the benefit of being able to be a bit vague in your query.

I tend to prefer the "ask the room" approach. But this isn't effective when you're working in different time zones, or at different hours. On the other hand, since software development is a collaborative activity, you'll need to address the question of how yo collaborate across time. (The simple answer is to try to avoid close coupling between activities that are likely to happen when the teams are on different schedules).

Situations like the one that I described can happen. But it's worth figuring out how often they happen, and if there are simple ways to reduce their incidence and impact. Good communication channels are importanty and sometimes the lower tech ones work better.

To Scrum, Prepare

noreply@blogger.com (Steve Berczuk) — Sun, 14 Nov 2010 20:30:00 +0000

Agile methods have some sort of daily all-team checkpoint meeting as part of the process. The idea behind the Daily Scrum (Scrum) or Daily Standup (XP) is good: replace status meetings (or someone walking around asking about status) with one short daily meeting where everyone has a chance to communicate about what they are doing and what they need help with. This ensures that there is at least one chance each day for everyone to understand the big picture of the project, and to discover unexpected dependencies.

But just having everyone in the room doesn't make for an effective, focused scrum. You need to be be prepared. Once I was on a team where the scrums started going off track. They took longer. People's updates were often "I don't remember what I did yesterday," or they became long unfocused rambles that didn't convey much information. I suggested that we all take a few minutes before Scrum to organize our thoughts. This got a lot of resistance. "It feels like a pre-meeting meeting, and with Scrum we're supposed to spend less time in meetings."

While Daily Scrum's are meant to be lightweight, it's respectful of everyone else's time to think about what 's worth sharing with the team. Most days you might just be working on one thing, in which case a quick glance at the Scrum board might be enough. But if you want to do what's best for your team, why not take 2 minutes before Scrum (either in the morning, or even the day before) jotting down what you want to share with the team that addresses the questions:

What did I do yesterday?
What do I plan to do today?
What were my roadblocks?

Starting each day with a clear picture in your head of the answers those questions is probably not a bad thing from a professional development perspective anyway.

Sure, everyone will have off days where they don't get around to this, but if your Scrum's are losing focus frequently, consider:

Posting a sign that reminds people of the agenda of the scrum (as Tabaka suggests in Collaboration Explained: Facilitation Skills for Software Project Leaders)
Whether there are people participating who aren't needed.
Whether your work isn't being structured in a way that moves the team towards the sprint goal.

The Daily Scrum (or standup) is a useful tool for being agile and responsive, but just being in the room does not mean that you are having a Scrum.