Reflections on Software Engineering

Why is Software Design Difficult?

2015-10-28T08:30:00.000-04:00

Software design requires a depth and breadth of experience that takes years to develop. While better training in software design might shorten the time required to master design by better codifying knowledge and experience for others to follow, attaining knowledge is easier said than done. Jonathan Mugan in his book The Curiosity Cycle explains:

Knowledge is more complicated than simply putting available pieces of information together because much of the world is hidden from view. [...] What we know determines what we see, therefore learning new things changes how we experience the world.

If it were a matter of obtaining some bit of information, then anyone could be a great designer after reading a book. Knowledge is an important part of design since it changes how you see the world, but perhaps even more important are the mental models we create to explain the world. These models are born through experience and seen through the lens of our current knowledge. Thinking about the psychology around how we build models is a fantastic way for us to understand software design - both generally and in specific instances. As in, "Why is design so hard?" but also "How do you 'do' design?"

The Curiosity Cycle

Design is all about learning. Learning about the problem, the solution, and where those two ideas meet. As children we learned to learn through the vehicle of curiosity.

The Curiosity Cycle from Mugan's book by the same name.

The cycle has three main building blocks.

Concepts - pieces (observations about elements and ideas) we pull from sensory experience. Elements are people, places, and things while an idea is something we think about and can put a boundary around.
Models - specify how concepts are put together to form useful information.
Tests - experiences that tell us how models work.

As an example, consider what you know about a door. A concept you might observe is that doors have doorknobs. A model you might build is "turning doorknobs opens doors." To test this model, you ask a question (form a hypothesis) and try it out. Maybe you find a doorknob that doesn't turn. Now you'll need to identify the concept of a locked door, update your model, and the cycle starts over again.

Individuating concepts is about pulling cohesive pieces out of the continuous whole experience. This is often an automatic event, but not always. Model building requires reflection on what you've observed and some abstraction. You need to come up with some way to describe what you've observed.

Tests are extremely important in this cycle. Tests can verify your model but also show (perhaps more importantly that your model is incomplete. An incomplete model is partially formed and does not reliably predict the environment. Incomplete models are extremely important because they help us to form new concepts. New concepts, in turn, help us to improve the models... and the cycle continues.

Curiosity-Driven Software Design

So why is software design so difficult? Consider this question from the context of the curiosity cycle.

Code is easy to see and most of us have direct experience writing, testing, running, and reading code. Therefore concepts from code are easy to discern through basic observation. Write enough code and think about it long enough and you'll discover many useful and interesting concepts.

Many design concepts are not easily observable in the artifacts. Early architectural design is pure thought stuff. Code, the most observable artifact, can easily obfuscate and often betrays intended architectural meaning. In addition, many models are dynamic or the combination of several structures (e.g. people + business + infrastructure + running system) and cannot be easily seen.

Many developers lack the necessary concepts to describe a model even when that model is a good one. For example, try to describe a car to someone who's never seen a car before. Is it a horseless carriage? Perhaps the Hyperloop is a modern example. "You know, it's one of those pneumatic tubes you'd find at the bank but for people!"

And yet, developing the models is still really important even though it's difficult. What makes this model building so difficult? I don't have a full answer for this, but here are some thoughts.

Inventing, observing, and describing concepts relevant to your system that can sometimes only exist as ideas is a challenge. This is required for design.
Simultaneously developing models of both the software structures and the domain in which the system lives is a challenge. Learning new ideas from either of these things changes the world view..
Testing models is difficult and also essential to successfully individuating concepts and forming a good enough model of the world to build something.
Asking good questions seems to be as much art as science. Some people are better at asking the right questions at the right times than others. This is unfortunate because asking questions is the fuel that keeps curiosity cycle turning.

The first few chapters of the Curiosity Cycle are available as a preview on Amazon Kindle. The primary purpose of the book is to help arm parents with a new perspective on how kids learn so they can help their kids develop a life long love of learning. While that's nice, I quickly lost interest (even as a new parent). All that said, the first few chapters are absolutely fascinating and the writing is extremely accessible with generally good references. Definitely check out the Kindle preview.

SATURN 2015 Highlights

2015-10-14T08:30:00.000-04:00

The year is nearly over and it seems safe to conclude that 2015 is the year software architecture went mainstream. Conferences that have been traditionally very technology heavy are including more and more software architecture design topics. O’Reilly Media and InfoQ added software architecture conferences to their line-ups this year and even Gartner named software architecture as a main theme for it’s 2015 Catalyst Conference.

As George Fairbanks and I were preparing our SATURN 2015 opening address it seemed fitting to reflect on SATURN’s 11 year journey and figure out where it fits in the evolving software architecture landscape that SATURN very much helped to create. During the SATURN 2015 opening address we shared what we see as SATURN’s main strengths. Three ideas that together we think have given SATURN it’s staying power and helped it remain relevant for over a decade.

Strong Foundation - Literacy is perhaps the most important factor to building a strong community. SATURN has always been dedicated to not only understanding and codifying foundational concepts in software architecture but also teaching those concepts to the next generation of designers.
Cutting Edge of Practice - For over a decade SATURN speakers have been pushing the state of practice forward. Stories from the field about what works and what doesn’t have made up the bulk of the technical program since the very first SATURN. SATURN is a place for sharing new ideas, getting feedback, and improving the way we practice software development.
Community - The SATURN conference brings together a group of like-minded people every year to geek out about the fundamentals and future of software architecture. The conference is small by design. We are a friendly crew and everyone is welcome. One of my favorite things about SATURN is that everyone is open to a good conversation, even the super popular authors and people with wikipedia pages.

Look through the SATURN archives and you’ll see that these three themes have permeated the conference since the beginning and in my opinion they continue to differentiate SATURN from other conferences. These are the SATURN community’s strengths and hopefully will make up the future for quite a while to come.

My SATURN 2015 Highlights

SATURN 2015 was a spectacular conference. Easily my favorite of the year. Here are some of my personal highlights with links to relevant talks, videos, and docs where appropriate.

Workshop on microservices architecture -- For the first time SATURN held an exploration style workshop as part of the conference. Under the guidance of Dennis Mancl (of OOPSLA and SPLASH workshop fame) George and I planned and facilitated a full day workshop where we asked tough questions and thought about different aspects of microservices architecture. The workshop agenda and outcomes are publicly available on GitHub.

Design Thinking is for you - Ari Font (IBM), Jeff Patton (Jeff Patton & Associates), and Jonathan Berger (Pivotal) hosted an amazing panel on design thinking. Architects are designers and good user interaction cuts deeply into a system, not just touching the surface UI. That said, architects have a lot to learn about collaborative design. I was so thankful that Ari, Jeff, and Jonathan accepted our invitation to speak at SATURN this year! Notes | Slides | Video

Design thinking office hours - SATURN 2015 Office hours were an open space style session with assigned moderators / hosts. Ari and Jonathan hosted a session on design thinking with about 8 people in attendance. For me this was one of the most valuable sessions of SATURN. I love exploring new ideas and the 8 or of us made some interesting discoveries in how software architecture design can become more aligned to design thinking as it’s currently practices.

Ari facilitating a lively and interactive discussion with some office hours participants.

New viewpoints for talking to stakeholders - This year’s Architecture in Practice award went to Jochem Schulenklopper and Eelco Rommes from Inspearit for the their talk "Why They Just Don’t Get it: Communicating Architecture to Business Stakeholders". This was such a refreshing and insightful talk, and I was so happy to see them get the peer selected Architecture in Practice award. The idea is something we talk about often but rarely do well -- vary the way you share information for your audience. Jocehm and Eelco shared several examples of ways they visualize architectural views in ways that are more interesting and obvious for business stakeholders’ concerns. Notes | Slides | Video

Updates on the Mini-QAW - In 2014, Will Chaparro and I presented a little method we used at IBM that was a trimmed down version of the Quality Attribute Workshop. We thought it was great, and the method had a huge response. What we didn’t expect was to see the method show up again, multiple times in SATURN 2015 talks by other architects who found our work, tried it, and found ways to make it even more awesome! Thijmen de Gooijer shared some particularly interesting and useful updates in his talk titled "Quality Requirements on a Shoestring" (slides).

Battledecks - I read about this silly thing on the internet and thought it might be fun as an evening social activity. Battledecks is an improvisation game in which a speaker is provided with a title and 10 slides they’ve never seen before, and asked to give a semi-coherent talk using the provided deck. I have never laughed as hard at a conference as I did during this session. Bravo SATURN for being willing to have some fun as a community!

Google’s talk, Perspectives on the Modern Practice of Software Architecture - Jeromey Carrier, Rick Buskens, and Jack Greenfield of Google shared some extremely insightful reflection and practical advice about some of their experiences at Google. What’s great about their stories from inside Google is that few companies are dealing with the scale and complexity they deal with. What’s also interesting about Google’s stories is that, at just over 17 years old, it seems like Google me be starting to feel their first significant pains from major legacy systems. I have a feeling there’s going to be even more interesting architecture stories from inside Google in the years to come.

My Silver Toolbox - With amazing Pecha Kucha talks from Gail Harris, Simon Brown, George Fairbanks, Ari Font, Eric Willeke, and Will Chaparro, this session was a blast and filled with many amazing nuggets of wisdom. What made this session even more special was that I turned the tables on the audience and managed to get even more nuggets of wisdom from them!

Participants in the "My Silver Toolbox" session brainstorming to decide
which software architecture tools are most important to them.
All the keynotes - Mary Shaw gave an awesome opening keynote that really set the stage for the conference. Gregor Hophe shared some amazing advice (which sadly can’t be released on video but some of his keynote is available as prose on his website). And Mark Schwartz shared some real and very candid stories about how he’s led US Citizenship and Immigration to modernize and really embrace the emerging, fast-paced, agile world that architect’s everywhere are responsible for fostering at their organizations. It was wonderful having such thinkers, leaders, and speakers share the stage at SATURN 2015.

Looking Forward to SATURN 2016

I had a blast at SATURN 2015 and not only had a chance to get crucial feedback on how I practice software architecture, but also learned many new ideas I've been able to use with my team at IBM. For me, it's a really great, inclusive, fun, smart community of people who are actively defining where software architecture design is headed. It was a privilege to be the custodian of this conference as the technical program chair for two years and I look forward to attending many more SATURN conferences as a regular attendee for years to come.

Compared with some of the other tech conferences out there, SATURN is not the flashiest conference, or the biggest, but as the longest running practitioner conference in software architecture, I think SATURN will continue to have a strong influence over what software architecture is and where it’s headed.

SATURN 2016 is now accepting proposals for 15, 30, and 90 minute talks. This conference is perfect for sharing new ideas and lessons from your experiences in software architecture design. Whether you're interested in design, technology, methods, businessy things, or working with teams, this is the place for ideas and stories that help us push the state of practice forward.

Three Big Ideas for Beating Innovator's Block

2015-10-01T08:30:00.000-04:00

I was invited to speak at the IBM dev@Insight conference last October. I just realized last week that the event was recorded and that the video is available online! My talk was about how my team overcame our "innovator's block" and created a set of integration patterns for using the IBM Watson Developer Cloud services within Watson Explorer applications. We see these patterns as a small start of something much bigger and more important. I hope that history sees our contributions as extremely minor and rather silly compared to the enormously awesome and amazing ideas that follow us in the coming months and years.
The talk is an Ignite talk so the slides auto-advance every 15 seconds. It makes for a fast, intense, information packed talk. I've also written a rough transcript of the talk for those who prefer to read (though the video is only five minutes long).

The Problem with Innovation

The phonautograuph was invented in 1857 by a Frenchman by the name of Edward-Leon Scott. What's interesting about the phonautograph is that it was the first ever sound recording device and it was a rather elegant invention for its day. The device works by picking up sound waves from the air, similarly to how the inner ear drum works and then transcribes the raw sound waves onto paper or smoked glass using a needle.

The most fascinating thing about the phonautograph is that there was no playback capability. Sound waves were written to paper much like music notes might be written to a page. Scott had envisioned a world in which trained professionals would some day be able to read the raw sound waves and be able to translate speech or music directly from the written wave form.

In retrospect, such a device is almost laughable but in context it is easy to forgive Scott for such an oversight as no playback. To start, this is the first device capable of sound recording, so that is an impressive feat. Next, up to that point the only means of capturing sound was through written forms such as sheet music for an opera or symphony. It was as if Scott was wearing innovation blinders based on his current context and physical reality. In the world he knew, people read sound from paper. It was certainly not created automatically from a recorded medium.

In the technology world we have a saying that summarizes this phenomena pretty well:

"I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail." (Abraham Maslow, 1966).

The important lesson here is that it is important to know how to use a variety of tools, otherwise your view on the world will be limited by your own capability or the context of your surroundings. Without a set of broad experiences you will be incapable of seeing different solutions to a given problem as everything will appear solvable only through the tools that you know how to use.

Search is my Hammer...

This was the problem that my team was facing when we were fist given the task of integrating the Watson Developer Cloud cognitive services with Watson Explorer. We know search and we know Watson Explorer extremely well. While we were aware of cloud application basics and many of the theories behind cognitive computing, we had no idea what to do with cloud-based cognitive services we were tasked with using, within the context of the search applications that we were used to deploying.

Focusing on the capabilities in the new technology we struggled to come up with good ways of using these new capabilities. And our first ideas were pretty bad.

"Maybe we could use Watson to 'cognitively enhance' text clustering?" we discussed. Text clustering, the ability to dynamically extract relevant themes from a set of search results such that it takes into account natural language including grammar (e.g. stemming, tense, useful vs. noise words, etc.) and knowledge (e.g. acronyms, synonyms, stop words, etc.), was one of Watson Explorer's original core capabilities born over a decade ago back when the product was still known as Vivisimo Velocity and Vivisimo was just a tiny startup. In and of itself this is not a terrible idea, it's just not super interesting.

"OK, well what about some kind of recommendation system like what Google and Bing do with ad words and user profiles?" This is getting warmer, but the more we talked it out, the less it made sense. How would this play out behind the firewall in an enterprise setting? Is this really the best we can do?

We have access to the most computing power and the most powerful algorithms in the history of the world and the best we can come up with is optimizing ads? Certainly we can do better than optimizing the consumerist machine!

Focus on the User

Success finally came when we stopped thinking about the technology and started focusing on the user. What problems did our users have? What kinds of things do they ussually do with Watson Explorer? How can we take solutions to those problems to the next level with the new tools that we had in the Watson Developer Cloud?

A common use case we see with Watson Explorer focuses on creating an "enhanced 360 degree view" of some entity. For example, you might look at a person entity and then from a single UI view search results pushed to you contextually based on metadata from a variety of sources, or computed analytics, or really any number of facets of that entity.

With a solid use case in hand, it was extremely easy to figure out wich of the cognitive services in the Watson Develper Cloud worked for our use cases and which ones did not. For example, Watson Personality Insights is a perfect complement to the 360 degree view of a person from either a real-time or pre-computed perspective. Question and Answer is a natural extension of search and might apply in a number of closely related use cases.

Rather than looking at our tools and figuring out what do with our hammers, we looked out into the world to understand what tools we needed first for the problems that were out there to be solved.

Play to your Strengths

My team is an engineering team. We're not product managers and not always super great at coming up with innovative, disruptive ideas. But we are extremely good at software design and development. It's what we do. Once we had some plausible use cases in hand we immediately shifted gears to bring the integration project into our wheelhouse: software development.

We immediately set about writing code and creating a useful architecture for how the Watson Developer Cloud might be used from Watson Explorer. The most important thing to emerge from this exercise (other than working examples) was a set of integration patterns for when and how you can use Watson Developer Cloud services from Watson Explorer. These patterns are important because it helps you to classify and quickly understand the nature of the solution once you know you're dealing with a certain kind of problem. Further, knowing the patterns offers another way to navigate the available services as not all Watson Developer Cloud services are applicable to all patterns.

Software architecture is one of our strengths. Once we fully embraced this, the project became much easier.

Collaborate

Reflecting on this project, one of the most valuable tools we had available to us was each other. Our best integration ideas and the best work we did were the a result of my team and IBM Watson as an organization coming together to collaborate. One of the most valuable activities we did was brainstorming and spit balling ideas on how Watson Developer Cloud services might work and what we could do with them. When we were working together it was fun and the results were so much better than working alone.

In the spirit of this collaboration we've released our sample code, documentation, and high-level ideas for use cases under an open source license on GitHub. Please take advantage of the work that we did. Build on what we started and create something truly amazing.

Let's Create Something Amazing

In 1878 Thomas Edison invented the phonograph, finally bringing playback capabilities - a full 20 years after Scott's phonautograph. The lesson here is that you have to start somewhere. Even if it seems silly. Even if it's not perfect. Even if all you know is that you aren't satisfied with the first steps you took. We're all in this together and defining the next era of computing is not going to happen by one person in a couple of months. It's going to take all of us working together, playing to our strengths and never letting our focus wander from creating something useful for people that helps them solve real problems.

Managing Multiple Ruby Versions with uru on Windows

2015-08-25T09:00:00.001-04:00

My undergraduate software engineering professor was awesome. Here's one bit of advice he shared during his course that really stuck with me over the years.

Pick a scripting language. Any Scripting Language. Learn to use it really, really well.

Compiled programming languages obviously have a purpose, but lightweight, interpreted scripting languages are an important tool as well. An interpreted scripting language is the perfect choice for one off tests, tedious text manipulation, automated tasks, and simple tools. Scripts are easy to write, easy to share, and good enough to get the job done.

My prof used Perl and was a whiz at it. I eventually settled on Ruby as my scripting language of choice. While the core language works really well across different platforms, as a predominantly Windows user, I find that Ruby sometimes runs into problems in odd situations. One thing that is particularly annoying is the lack of a Ruby version manager. Version managers are a great idea. The basic idea is to make it easier to switch between different Ruby versions on the command line so that it's easier to test and develop against multiple projects.

OS X and Linux users get RVM. Until recently, the only option on Windows was pik, a woefully out of date and problematic open source project that is no longer under active development (I'm not going to even bother linking to it). Thankfully, now there's uru, a cross platform Ruby environment manager that works on Windows, Linux, and OS X.

I've been using uru for close to a year now and it's worked out very well for me. That said, when first getting uru up and running I did run into a few snags. Here's the steps I used to overcome those problems. If you're trying to run multiple Ruby versions on Windows you should give uru a try.

Step 1: Install uru

First, download and install uru. To do this, download the zip and unpack it to a place on your path. I have a generic "tools" folder for things like this. Next, follow the instructions on the uru website. Basically, just run...

C:\tools> uru_rt admin install

Step 2: Install Ruby

Next, download and install the Ruby versions you want. You may have up to 1 "system" Ruby when using uru. If you choose to have a system Ruby then that should be the only one you allow to be added to the path. I installed different versions of Ruby using the Ruby Windows Installer and also the latest JRuby. I don't have any Ruby versions on my path and instead choose the Ruby version I want to use explicitly each time. The main thing is to just be careful about what gets added to your path when performing the installations.

Step 3: Register your Ruby Installations with uru

By this point I have installed all of my Ruby versions to the C:\ruby\ directory with none on the path. The next step is to register each Ruby version using the uru command line.

C:\> uru admin add c:\ruby\jruby-1.7.18\bin\
---> Registered jruby at `c:\ruby\jruby-1.7.18\bin` as `1718`

C:\> uru ls
    1718        : jruby 1.7.18 (1.9.3p551) 2014-12-22 625381c on Java HotSpot(TM) ...
    193         : ruby 1.9.3p551 (2014-11-13) [i386-mingw32]
    215         : ruby 2.1.5p273 (2014-11-13 revision 48405) [x64-mingw32]

Once installed I changed the tag associated with JRuby to be a little easier to remember.

C:\> uru admin retag 1718 jruby
---> retagged `1718` to `jruby`

At this point, I'm up and running. Now switching my environment to use JRuby is as simple as...

C:\> uru jruby
---> Now using jruby 1.7.18 tagged as `jruby`

C:\> jruby --version
jruby 1.7.18 (1.9.3p551) 2014-12-22 625381c on Java HotSpot(TM) Client VM 1.8.0_31-b13 +jit [Windows 7-x86]

C:\> ruby --version
'ruby' is not recognized as an internal or external command,
operable program or batch file.

C:\> uru 193
---> Now using ruby 1.9.3-p551 tagged as `193`

C:\> jruby --version
'jruby' is not recognized as an internal or external command,
operable program or batch file.

C:\> ruby --version
ruby 1.9.3p551 (2014-11-13) [i386-mingw32]

Step 4: Install Ruby Windows DevKit

The last step I took was installing the Ruby Windows DevKit. You'll need this to install any gems that require native extensions such as json. DevKit is just an archive that you unzip to a specific location and run an install script to copy some files into the right places so Ruby can use them.

I generally store the DevKit next to the version of Ruby to which it belongs, for example, C:\ruby\Ruby193\devkit where Ruby193 is the folder containing the 193 version of Ruby that I registered with uru. JRuby does not need a DevKit. Just follow the normal instructions for installing DevKit and install it for each of the Ruby versions you want to use, obviously using the version of Ruby for which you are installing DevKit.

As a last step, install the json gem to make sure DevKit is installed correctly...

C:\> gem install json
ERROR:  Could not find a valid gem 'json', here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 
          state=SSLv3 read server certificate B: certificate verify failed ....

Oh no! SSL Hell! Apparently rubygems.org recently updated their SSL Certificates which requires clients to also update. This error comes up because the security certificates used by gem are out of date. Thankfully it's pretty easy to update the certs.

To fix this, follow the instructions on this StackOverflow question: SSL Error When installing rubygems, Unable to pull data from 'https://rubygems.org/ You'll need to download the updated certificate and manually install it in each Gem directory....

C:\> uru 193
C:\> gem which rubygems
C:/ruby/Ruby193/lib/ruby/site_ruby/1.9.1/rubygems.rb

With all that completed, everything should now work as expected.

C:\> gem install json
Temporarily enhancing PATH to include DevKit...
Building native extensions.  This could take a while...
Successfully installed json-1.8.2
Parsing documentation for json-1.8.2
Done installing documentation for json after 0 seconds
1 gem installed

Step 5: Enjoy Hacking in Ruby

My workmates tell me I just need to switch to Linux already. I do run a very Linux-like Windows environment with most of the GNU command line tools available from the Windows command line. Whatevs. I like my Windows and with uru at least I can switch between different Rubies with relative ease.

Whether it's Ruby or something else, pick a scripting language, any scripting language, and learn it well. If you're running Ruby on Windows, hopefully this little guide helped you skip some frustrations!

Introduction to Microservices Architecture

2015-05-20T08:30:00.000-04:00

I recently returned from the O'Reilly Software Architecture conference where I presented/facilitated a workshop on Rapid Software Architecture Exploration. My workshop went great. According to the reviews it was among the better presentations at the conference -- which is awesome! But that's not what this blog post is about.

Microservices architecture was a significant part of the conference with well over 50% of the sessions dedicated to the topic. And I'm being really nice in saying it was only 50% - I think it was closer to 90% and perhaps could have been called the O'Reilly Microservices Conference. Anyway, so there's a ton of hype on this topic. And since we're looking closely at the (loosely defined) microservices architectural style for an upcoming project at work, this was a great learning opportunity. I am currently reading the book Building Microservices by Sam Newman but I thought it would be useful to share some of the insights I gleaned from the conference now rather than waiting until I've finished the book.

"And you are now an anointed solider in the microservices army..."

What are Microservices?

Adrian Cockcroft (of Netflix fame) defines microservices as the following.

Microservices are a loosely coupled Service-Oriented Architecture (SOA) with bounded contexts.

This definition has a couple of parts that are worth further exploration.

Loosely Coupled implies that services are independently updateable units. Ultimately this means that updating or deploying one service should not require any modifications to other services in the architecture. When multiple updates are required this could mean that the partitioning is not quite right and that perhaps services should be refactored to removing coupling or perhaps even combined.
SOA has been around for more than a decade and many people have commented that microservices architecture is really just "SOA done right". Many of the ideas you may already be familiar with from traditional SOA still apply -- for example, when dealing with data and distributed web-based systems. Some lessons are also out of date, probably the most obvious being that SOAP has gone out of style in favor of lighter and simpler transport mechanisms. To me much of it just seems a matter of technology fashion.
Bounded contexts comes from Domain Driven Design by Eric Evans. Bounded context requires that a domain model is internally consistent within the context of an explicitly established boundary. In microservices architecture, the boundary for the domain becomes the public API through which all external entities communicate with the microservice. With these explicit boundaries then each microservice represents a partitioning of the whole domain model. The image below from Martin Fowler captures the idea of bounded context fairly well.

Bounded context partitions a model into multiple contexts, each with clearly defined boundaries.
Within the context, the model is self-consistent.

This idea that each microservice embodies a bounded context is actually pretty important and a key to what makes microservices architecture work.

When the domain model is properly understood and partitioned, this enables the loose coupling among services.
APIs are derived from boundaries within the whole model.
Each service focuses on only one (or very few) ideas. Even the example above could potentially be considered too big by some.
Smaller models are easier to understand, design, code, and test. As a rule of thumb, the context for a service (and the code that implements that context) should easily fit into 1 or 2 developer's memory. An alternative rule of thumb is that a microservice can be completed, tested, and shippable ("done done") in 1-2 weeks time.

Perhaps the most interesting implication of bounded context from the perspective of microservices is that it enables self-organization which is essential for any microservices architecture to succeed. As long as explicit boundaries exist among the domain model, and as long as partitions of the model are self-consistent, an organization can self-organize around domain concepts, which in turn allows software systems that organization builds to evolve to match the current understanding of the domain. Microservices proponents call this "Reversing Conway's Law" and the concept was observed by Fred Brooks in the Mythical Man Month years ago.

[Conway's Law states that] Organizations that design systems are constrained to produce copies of the communication structures of these organizations [...] the organization chart will initially reflect the first system design, which is almost surely not the right one [...] as one learns, he changes the design [...]. Management structures also need to be changed as the system changes...

Therefore, individuals must be allowed to self-organize so that they can evolve the architecture design as necessary. Within this loosely coupled world of bounded contexts in which only one or two people represent the entirety of a single shippable unit, individuals need to be empowered to take initiative and do what is right. The gist is that if you want a loosely coupled system with bounded context then these ideals must be embodied in the teams teams that build such systems.

Blah blah microservices blah blah bounded context blah blah Conway's law. After a while that's all it sounds like.

There is a ton of hype around this topic almost to the point of absurdity. My gut reaction to all of this is that it's interesting and great to see people talking about modeling, organizational structure, agility, modular design, and useful-but-seemingly-forgotten software engineering fundamentals again -- but I can't help but see microservices as just really excellent packaging for the same old ideas so the consultants and tool vendors can have something new to sell.

After really digging into the literature, I'm not willing to cynically dismiss microservices, but I do hold a healthy bit of skepticism as I get my hands dirty building my first fine-grained, distributed, service-oriented components. Let's say, I'm optimistically hopeful for now and it all seems very interesting but I need to gain more first hand experience.

Communicating Design Intent by Sharing the Paths not Taken

2015-03-31T09:00:00.000-04:00

If you're even remotely a movie buff then you'll probably really enjoy Tony Zhou's "Every Frame a Painting" series. His analytical, accessible, and entertaining film studies mini-documentaries are wonderful. Each video in the series focuses on one main idea, often something overlooked or perhaps under appreciated by other films studies enthusiasts (analysts? critics?). This bit of analysis on what makes Jackie Chan's physical comedy so brilliant is... well, itself brilliantly done.

In this episode of Every Frame a Painting, Tony discusses the idea of showing character choice rather than just talking about it, using Snowpiercer as an example:

In nearly every blockbuster you've watched or tell-tale game that you've played, you've seen moments like this: the protagonist has to make a choice and there's no way to reverse it.

Just to be clear, I think moments like these are great and a foundation of good storytelling but I do wish in movies they weren't presented through dialog. So today I look at one possible alternative. In fact it's one of the oldest traditions in cinema. How do you show character choice? Left or right. That's it.

In a similar fashion as many blockbuster and video game protagonists software architects regularly encounter complex choices that often can't be easily reversed. Or rather, the architecture we're designing experiences those complex choices (since in this metaphor we're working behind the camera and the architecture is the character on screen). But, unlike blockbusters and video games, software architects don't have the luxury of an audience that can simply watch the architecture evolve with each design decision made.

Even on small teams, generally only a few people at a time are in the room or at the whiteboard when a key decision about the architecture is made. In some ways it's as if we are asking downstream designers to watch only the last 10 minutes of Snowpiercer and then try to figure out why it's such a big deal for Chris Evan's character, Curtis, to do what he does. Without the context of these critical decision points that are constantly happening throughout a character's/architecture's journey, the audience/downstream designer loses significant rationale that could help explain a lot.

Are there methods that architects can use to make architectural choice evident? The answer to this question is certainly, "Yes!"

The Paths Not Taken

While downstream designers can't watch the architecture make a choice on the silver screen (left or right?), we can still make it easier for developers who read our documents to playback the architecture's "character development" more easily.

Sometimes when trying to communicate design intent and rationale, it's more effective to say what you didn't do rather than what you chose to do. The method I'm proposing is pretty simple. Create a list of the architectural decisions you discarded with a brief note for why those decisions were rejected. With this list in hand, downstream designers have an opportunity to replay critical decision points and can now watch the architecture develop and mature. Left or right? What critical moments did this architecture face and what path did it end up taking?

Here's an example from a recent project. The basic problem involves figuring out how to integrate our client software with third-party, cloud-based web services that application developers using our software are starting to employ more frequently. The specifics aren't too important for the example, and I can't really share the specifics anyway.

Path not Taken	Why this Path was Rejected
Create a cloud-based "services adapter" to buffer against changes in third-party services.	New component that must be maintained by the team. This increases the workload for development, QA, docs, ... everyone. Benefits in features and quality attributes are not required for MVP release Costs significantly outweigh the benefits.
Release adapter as open source, have customers load it themselves	Extra steps to deployment inhibit adoption Open source is great, but unmodified defaults could mean security issues down the road. Worries about training for customers and delivery consultants New component that must be maintained by the team
Offer a client library that makes it easier for developers to connect to third party services directly	Client library will always be out of date (services are cloud-based with continuous delivery, our software is not) Increased learning curve in that customers must learn both how to use the service and how to use the client library. Documentation overhead - client library must be fully documented. Design overhead, client library becomes a stable API... high risk in getting it right the first time on short schedule. New component, increases workload for team
No new support for web services integration in the client software	Doing nothing does not improve the user experience New patterns and paradigms are emerging that require some kind of help be available to application developers.

With this background and context, its' pretty easy to see that the design choices we made kept our architecture on a path toward simplicity and tended to favor less code over more code. With this history and context, the list of design decisions that were rejected, it's much easier to understand why the resulting architecture is what it is. You're essentially replaying the architecture's definable moments. Hopefully with this information downstream developers won't scratch their heads when they see what might appear to be an odd outcome for the architecture/character. They can see for themselves the key decisions that led to the architecture/character becoming what it became and in turn internalize a significantly more nuanced appreciation for the architecture/character as a whole.

Again, the insightful Tony Zhou:

So, the next time you your character runs into a complex choice, who knows? Maybe it's actually just a simple matter of: left or right?

Agile Software Architecture (with George Fairbanks)

2015-03-17T08:30:00.000-04:00

George Fairbanks and I chatted via Google+ Hangout in February 2014 about agile software architecture. I think we found a groove pretty quickly and the conversation overall went great. I've created bookmarks of some of the main topic areas that we covered to make it easier to jump to bits of the discussion you might find more interesting.

During this Hangout we meandered through several topics starting with walking skeletons and leading to...

Walking skeletons, thin vertical slices, and assumptions about "ideal" patterns for agile architecture
YAGNI and how you figure out what the bones of a walking skeleton should be
System Metaphor
Design Patterns

Presumptive architectures
Routine vs. novel design
Including the "just like the last thing we built" pattern
How patterns mature over time... and a seeming lack of attention to patterns these days

I proposed that perhaps patterns are being codified as frameworks (and not patterns catalogs)

Choose a framework, architecture chosen
Balancing flexibility and power when choosing frameworks

Switching frameworks is really hard...

Refactoring in the small vs. refactoring in the large
Can you refactor to patterns and back your way into a framework? Maybe....
If you choose the wrong framework you might be stuck

Last responsible moment

There's a tension in keeping options open since it's so expensive (or impossible) to change frameworks later.

Spiral model

Great model for thinking about up front design decisions in the right ways.
Also, correct pronunciation of Boehm, ;-)

Michael Waterman's research -- see also, his research paper and blog.

Strategically deferring architectural decisions
Trading speed of delivery for knowledge - "That would be a nice problem to have"
See also Boehm's and Turner's book Balancing Agility and Discipline: A Guide for the Perplexed
Does it make sense to choose a framework that promotes properties you need today to deliver faster at the sacrifice of properties you might need later?

Engineering by design vs. Engineering by coincidence

Choosing architectural mismatch to promote different qualities?

Architecture Haiku

2015-03-03T08:30:00.000-05:00

The most effective way to transfer complex, abstract ideas from one person’s brain to another person’s brain is by writing the ideas down. This premise permeates our whole approach to software design. Entire books are dedicated to creating better documentation! Following traditional documentation advice inevitably leads to comprehensive, albeit verbose, documents filled with important details that, sadly, too few stakeholders read. There must be a better way.

Introducing the Architecture Haiku

George Fairbanks originally came up with the idea of the Architecture Haiku as "a one-page, quick-to-build, uber-terse design description." With only one page of paper (assumed standard, A4) to work with there is no space to be indecisive or indirect. There might not even be room for diagrams - which at first sounds crazy, but perhaps is crazy brilliant! We tried this idea on my team and it turned out to be a great, lightweight method for writing down and sharing architecture decisions.

With so little room, picking the right information to include is extremely important. Here's what Fairbanks recommends:

a brief summary of the overall solution,
a list of important technical constraints,
a high-level summary of key functional requirements,
a prioritized list of quality attributes,
a brief explanation of design decisions including rationale and trade-offs,
a list of architectural styles and patterns used, and
only those diagrams that add meaning beyond the information already on the page.

Not including diagrams seems kinda nuts, after all software architects are famous (infamous?) for their box and line diagrams. But there is power in language and an extensive pattern catalog has been built up around software design. When existing architectural styles and patterns can't quite describe an idea, home grown system metaphors can fill the gap.

Is one page really enough? We found that one page is more than enough, but it takes some thought. The intent is not to capture all possible information but rather to highlight the most critical ideas.

The Architecture Haiku then plays two important roles. First, it creates a vision of the architecture that can be easily consumed and shared. Second, this single page of documentation records cognitive landmarks and other reference points that help teammates recall prior decisions, stories, whiteboard sessions, conversations, and so on. On the path to discovering the architecture for the software system you're going to build, your team develops a shared understanding for much of the design. Haiku is a fast and easy way to record key decisions and landmarks for navigating your past discussions, memories, and decisions.

Example Architecture Haiku, scanned from a real project.
Notice the hand written annotations. You can say a lot in only one page.

Tips for using Architecture Haiku

In using the Architecture Haiku with my team, we discovered several tips for effectively creating them.

Establish a common software architecture vocabulary for your team. It's easier to communicate effectively when everyone speaks the same language and has the same conceptual understanding.
Treat the Architecture Haiku as a living document. As you learn more or as decisions change, update the document. It's only one page so this is easy to do.
Explore the design space first, then record decisions in Haiku. To be concise you have to know what's important. This takes a little work, but again if you treat the Haiku as a living document then it becomes a tool to help you unlike longer documents which can become a burden.
Use a common template and good graphic design practices. Making the font tiny just to fit everything on one page defeats the purpose of creating an easily consumable reference document. The true goal of the Architecture Haiku is to be concise and force difficult decisions that enhance clarity.
Use the architecture Haiku as an outline for future documentation. Once you know what's really important, use the Haiku to help guide the creation of other documents. In some cases you can treat it as an outline or even as an executive summary for a more fully realized architecture description document.

An Architecture Haiku Template!

To help you get started with using the Architecture Haiku with your team, here is a simple template that includes... wait for it.... a one page description of the Architecture Haiku!

Some other resources on Architecture Haiku and related topics.

George Fairbanks. Architecture Haiku. Includes slides from the original presentations as well as write-ups on the Architecture Haiku.
Michael Keeling. Creating an Architecture Oral History: Minimalist Techniques for Describing Systems. SATURN 2012. This talk summarizes my team's experience using Architecture Haiku and explores the idea of an architecture oral history that is fostered among a team.

Give it a try! We've had a lot of success with Architecture Haiku and I think you will too. Send me an email or leave a comment to let me know how it goes. I think it would be awesome if we had a collection of Haiku examples from different kinds of software systems and open source projects.

Dante can be Agile too

2014-12-03T08:30:00.000-05:00

[I recently gave a talk for the monthly SEI Agile Collaboration Meeting. The Agile Collaboration Meeting brings together upwards of 50 or so agile practitioners from throughout the federal and DoD space including civilian contractors. This post is a summary of the talk I gave at the September meeting.]

When you do a search for pictures of "unrequited love" one of the first images that comes up is this 1884 painting by Henry Holiday titled "Dante and Beatrice." As Beatrice and Dante were Italians it is extremely important that you pronounce her name in your mind correctly – Bee-a-tree-che – using the Italian pronunciation. In this painting you see Dante, the poet of the Divine Comedy among many other works, chilling on a bridge as he is snubbed by a posse of three women. Beatrice is the one in yellow completely ignoring Dante, who probably went out of his way to be in the right place at the right time so he could see her.

In real life, Dante was obsessed with Beatrice. They met when they were children and their families cruised in many of the same social and professional circles. Allegedly it was love at first sight for Dante, then age 9. Both Dante and Beatrice married other people and apparently met only one other time before Beatrice died at the young age of 24. Dante wrote of Beatrice as "his savior" and she was a prominent character in his Vita Nova (a collection of poems professing his love for both the actual and a metaphorical Beatrice) and Divine Comedy (in which she escorts Dante through Heaven).

I propose that there is an unrequited love between the software architecture and agile communities; software architecture is the Dante to an agile Beatrice who simply ignores him. Certainly if this were true there would be some observable facts to either support or refute this idea.

Dante and Beatrice, painting by Henry Holiday (1884)

Hypothesis: The Software Architecture Community "Loves" Agile more than the Agile Community "Loves" Software Architecture

To test this hypothesis I will examine submissions from popular agile and software architecture conferences. The two proxies I will use are the SATURN conference representing the software architecture community, and the national Agile conference (sometimes called "The Big Agile Conference") representing the agile community. Both conferences are about 10 years old and are practitioner focused events that attract a good mix of working professionals and thought leaders.

The Agile conference is about 10x larger than SATURN both in terms of size and number of submissions. That being said both conferences have a similar peer review process in which experts from the community review and recommend submissions for inclusion in the program and both conferences are focused on practice opposed to academic research.

My hypothesis is that the software architecture community "loves" agile more than the agile community "loves" software architecture. The implication being that architectures "love" for agile is not returned, or unrequited. I would expect this to be evident in the session topics presented at each conference.

Testing Method: Conference Sessions Analysis

Love for a topic might be shown by the number of talks that are accepted on that topic at each conference. For example, the presence of a software architecture related talk at the Agile conference would indicate a +1 for Agile's love of software architecture.

I looked at the schedules for all Agile and SATURN conferences since 2011. For each I used "expert judgment" to determine, based on the title, abstract, and speaker, whether the topic might be relevant in a different community. In the Agile schedules I was looking for software architecture talks. In the SATURN schedules I was looking for agile topics.

I'll fully admit that a single data point of "expert judgment" is not the most rigorous test, but it's a start. Am I an expert? I have been a regular speaker and attendee at various agile and software development and design conferences since 2009. I've attended both SATURN and Agile twice in this time frame. I've helped with the review committee for Agile experience reports since 2012 and SATURN since 2013. I was the technical chair for SATURN in 2014 and I am a technical co-chair for SATURN 2015. I also have a Master in Software Engineering from Carnegie Mellon and have studied and practiced both agile and software architecture fairly extensively in my career as a software engineer so far. It's not perfect, but I have enough perspective for the analysis I hope.

For the Agile conference I was looking for topics that were directly related to software architecture. The basic rubric was anything related to the following topics (and I tried to be really liberal in what I included).

Software architecture – include methods, notations, design, and lessons learned kinds of talks
Technology – how things work and why and when you'd use them
Technical debt
Modeling or enterprise architecture
Anything that dealt with requirements beyond just user stores, features, and functional requirements – I was looking for broader thought about system quality attributes.

For the SATURN conference I was looking for topics that were directly related to agile software development methods. I tended to be a little stricter on the SATURN sessions mostly because I felt that "agile" was a generally broader topic than architecture.

Lightweight or lean methods – things that might not have come out and said "I'm agile" but were definitely in the spirit of agility.
Architecture in agile context – for example talking about an agile team or experience
Continuous delivery – both technology and value
Technical debt

Again, the basic question I asked is, "Would this session work at the other conference?" If yes, it was a +1 for that community's love of the other.

Results of the Analysis

Looking at raw session counts, the two conferences seem about even in the time frame I examined with about 30 sessions at the Agile Conference that would likely work well at SATURN and about 35 sessions at SATURN that would likely do well at the Agile conference. I know for a fact that at least a few of the sessions definitely fit in this category since there are few repeat presenters including myself, Rebecca Wirfs-Brock, Andy Hunt, and Joe Justice among others.

	Agile	SATURN
2011	10	7
2012	6	8
2013	9	9
2014	5	11
Total	30	35

Looking at the percentages paints a completely different picture. Close to a quarter – 22% – of the talks at SATURN are agile-related while barely 5% of talks at Agile deal with software architecture related topics. The Agile Conference consistently had about 145 sessions in the schedule while SATURN had about 35 sessions in 2011 and 2012 and 45 sessions in 2012 and 2014. Using these approximate totals with the previous raw counts table gives the following percentage of sessions that might fit in at the other conference.

	Agile	SATURN
2011	7%	21%
2012	4%	18%
2013	6%	26%
2014	3%	24%
Total	5%	22%

This seems like pretty overwhelming evidence in support of the hypothesis. As a side note, this result is definitely dependent on the proxy used. For example, I haven't done the analysis but my experience with the XP conference in Europe and the European Agile community in general is that they have a much greater focus on architecture and design than in the US.

Another interesting data point in support of the hypothesis is the SATURN "Architecture in Practice" Best Presentation award. Interestingly, 4 out of 5 of the awards have gone to presentations on agile or lean topics since the award was introduced in 2010.

There really isn't an equivalent award at the Agile Conference that spans across nearly all sessions like it does at SATURN, but this is still an interesting data point. The SATURN "Architecture in Practice" award is "awarded to the presentation that best describes experiences, methods, and lessons learned from the implementation of architecture-centric practices." It's a peer voted award in which the attendees decide the winner via voting cards turned in just before the final keynote at the conference.

The hypothesis is generally supported (with plenty of caveats and room for debate)... but the more interesting question might be: Why does this unrequited love exists in the first place?

From whence doth this apparent unrequited love arise?

To understand this question we have to understand a little more about how software is developed and the roles that agility and architecture play in software development.

Consider a continuum of knowledge. If I were to graph this (see below), the higher on the continuum you are, the more vision-oriented thinking you're doing. Meanwhile the lower on the continuum the more detail-oriented you think. An easy way to look at this is that the CEO and product management types are dealing with very coarse grained abstractions at the conceptual idea level toward the top, while programmers are dealing with much finer grained abstractions at the detailed level toward the bottom.

Ideas are easier to change and must eventually become something more detailed and concrete
which in turn becomes more difficult to change. Kind of "duh" but the model makes some
interesting implications a little more obvious.

At some point to build some software you have to cross over a theoretical "realization boundary" in which your ideas are translated into reality. A vision eventually becomes code. An interesting implication of this continuum is that the higher up you are (idea stuff) the easier it is to change while the lower down (concrete stuff) the more difficult it is to change. So in some ways, the foundation of what will become reality is built upon the stuff that is easiest to change.

This shouldn't be too surprising, but given that the stuff of dreams (the vision for a software system) essentially form the foundation for what eventually gets built, you create a situation where you almost have a house built on sand. The visionary leader lives in a world that is easy to change and for reasons both silly and mission critical, this means that the software we build must be easily changeable too. Miss the boat on this and you risk the ire of your visionary at best, becoming irrelevant and missing business opportunities at worst.

A "visionary leader" (CEO or Product Management or others for example) can change their minds easily
for a variety of silly and very legitimate reasons. Can your architecture stand up to the change?

It's almost as if we've created a house built on sand. And indeed a hot and valuable topic that comes up often is creating pliable, malleable architectures so you can respond to change more easily.

Embracing Change is the name of the game

Software is built easily changeable premises and I will argue that as frustrating as this can be, it's actually a really, really good thing. It seems then that the implication of all of this is that enabling agility is the whole point of software architecture. The key to this lies in understanding how to cross that "realization boundary".

A central focus of software architecture is in understanding how to transition the "realization boundary" so that the vision can become real software. This makes sense because software architects are required to live in both the vision-oriented and detail-oriented worlds. It is the responsibility of the architect to chaperon the vision from the minds of stakeholders to the fingers of the programmers.

To characterize very broadly, Agile generally does a great job dealing with either vision-oriented stuff or detail-oriented stuff, but rarely addresses crossing this realization boundary. It's like that comic about the mathematician who does some hand waving -- "a miracle appears here" -- in the middle of a complex proof to arrive at a solution.

The insights surfaced by this vision-to-detail model is interesting, but it still doesn't explain exactly why this might be happening. If embracing change is the point, and if software architecture strives to support agility, then why is architecture's love still unrequited?

Dante (Software Architecture): "Hi, Agile, I'm ready to embrace change!"
Beatrice (Agile): *sigh* "It's that software architecture again... he's just so... structured." /eyeroll

The answer in this case may come down to the nature of agile and architecture. Agile software development promotes preserving choice while software architecture promotes preserving structure. In many ways there is a natural tension in these two ideas, even when you are trying to preserve structure that preserves choice (head spinningly meta, I know).

Ultimately, for any team, a lot of things have to go "just right" for a team to be agile. It's more than just the methods they use, it comes down to culture and perhaps even grit. As an agile software architect, I believe that having a solid understanding of software architecture design is one of the more important and interesting things required for everything to go "just right" so a team can be agile. There are other philosophies as well, and the best part is that they all can work in the right circumstances.

SATURN 2015 is an Great Conference for Agile Architects

I've said it before and here it is again, XP is a great conference. The "big" Agile conference is pretty good too. But SATURN is probably the best conference I've seen for software architects. If you are a software architect or interested in software architecture and anyway involved with an agile project, then SATURN is probably a better learning and networking event for you than the Agile conference. For these topics, the signal to noise ratio is very good. The conference itself is relatively small which means more meaningful network and greater engagement. There is just an all around higher density of software architecture information.

And what's better than attending? Presenting! The submissions system is open now and proposals are being reviewed and accepted on a continuous basis through January 16, 2015. Submit early to improve your chances of acceptance. Submit as many talks as you want.

SATURN is a great conference for agile architects and I hope to see you in Baltimore this April!

SATURN Conference: An Agile Practitioner's Perspective

The Case for Minimalism in Software Architecture

2014-11-06T08:30:00.000-05:00

"Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupery

I was first introduced to the idea of minimalist software architecture in an article by Ruth Malan and Dana Bredemeyer, "Less is More with Minimalist Architecture" (PDF). The basic premise is devilishly simple: the software architect's job is to "only do with architecture what is absolutely necessary to achieve key, broadly scoped qualities of your system." Any design decisions that do not support this goal are left to the discretion of downstream designers.

In my experience, minimalism defined in this way is a powerful tool because it helps me keep my attention squarely on system qualities. I've found that focusing only on the most important quality attributes creates a clear delineation between "architectural" design and "detailed" design. As a result, satisfying quality attributes becomes a signal for stopping architectural design activities and helps me avoid waste through over-analysis.

Even more importantly, minimalist designs provide critical guidance without removing autonomy from downstream designers. Who are these downstream designers? In my experience with small teams, it is myself and my teammates now playing the role of programmers. In larger groups, entire other teams of people that I sometimes have never met will fill this role and take responsibility for developing and detailing coarse grained components.

The autonomy afforded by minimalism is critical for allowing team self-organization, which in turn allows for better lower-level designs. Fred Brooks summarizes the reasoning for this phenomenon best in his essay "The Documentary Hypothesis" from The Mythical Man-Month. Conway's Law predicts that a system's design is destined to reflect the communication structure of the team that built it; therefore a team's organization must change for the system's design to change. Minimalism reduces friction of change which in turn makes it easier to self-organize which in turn helps you make better designs.

While minimalist software architecture makes sense, should the documentation necessarily be minimalist as well? The answer, of course, is that it depends. But, in my experience, creating a minimalist software architecture description encourages me to also design a minimalist architecture for the system documented. It seems that by forcing myself to use a minimalist packaging, my only choice is to focus on the most important aspects of the system's design.

Another keenly positive result of having tightly focused, minimalist documentation is that I am able to more quickly communicate essential design decisions to my team. Important decisions and rationale are distilled to a pithy phrase or two that can be easily shared and remembered. And change is easier too. Modifying a design, in the case of an Architecture Haiku or System Metaphor, is as simple as scratching out a few words on paper or adding an annotation, or changing the stories you share.

Gone is the 30 page (or 50 or 100 page!) architecture description that my team ignores anyway. Replacing it is a flyer advertising the key design decisions in the system we're building; a constant reminder of those decisions, their rationale, and the context around them. As a result, minimalist software architecture descriptions serve as an effective catalyst for meaningful discussion and a practical guidebook for implementation.

Untitled (Blue Leaning Plank) by John McCracken at the Carnegie Museum of Art.
A great example of minimalism and one of my favorite sculptures at this museum.
My wife thinks I'm nuts for digging this piece soooo much. This picture barely does it justice.

Defect-Driven Process Improvement

2014-10-29T08:30:00.000-04:00

Defects can tell you a lot about your team. And not just what the problems are in the code. The Team Software Process uses a very simple method of defect classification to unlock hidden insights. My team recently did some analysis on our bug database at the conclusion of a project and unearthed some surprising information about how we deal with requirements as well as some weaknesses we need to work on.

Simple Defect Classification

The simplest defect classification method involves manually evaluating defects to add contextual meta information. The Team Software Process suggests some of the following pieces of information as a starting point for creating a defect taxonomy.

Related Feature - the feature to which the defect was reported against
Module Discovered - the code in which the defect was found
Defect Type - a standardized description of the defect, such as "assignment," "data," or "syntax."
Reason Injected - why did the bug make it into the software in the first place? This should also be standardized, for example "education," "oversight," or "mistake."
Phase Injected - what were the activities being performed when the defect was injected? For example, "requirements," "design," or "implementation."
Phase Discovered - what were the activities being performed when the defect was discovered? These would be the same as the list for phase injected.

You'll recognize this simple defect classification method from the Square Root team's pair programming experiments. Ideally this classification method can also be built directly into your defect tracker.

This simple method has some disadvantages, the biggest one being that you usually will not be able to understand the reason a defect was injected until after you understand the root cause of the defect. Also, unless you were the one who wrote the code, you can only guess at the reason (was this an oversight, mistake, or education?) the defect was injected. As such, some knowledge of the system is required to perform the analysis. There are other methods for defect classification that overcome this problem but take a little more training and planning to apply well. One example is the Orthogonal Defect Classification method from IBM Research.

Since the defects in a software system are a direct reflection of the process, knowledge, and skills of the team that injected them, once the defects are classified you have some serious power for enacting positive change.

A Snapshot of our Defect Data

The following graphs summarize some of the more interesting trends we discovered by looking at our data. There were two very interesting findings. First, we had a major problem with requirements that resulted in significant rework. Second, we need to rethink the way we test and work to improve.

Figure 1 shows the reason why various defects were injected. Defects that originate in tests were excluded from this particular analysis so that we could focus on other areas that might also be interesting. Communication in our vernacular refers to cases where the defect resulted from misinformation originating in documentation or personal interactions (e.g. a meeting). We noticed that a disproportionately large number of defects resulted from miscommunication and so broke down communication into true communication problems, changes that were not effectively communicated, and new requirements that were only flushed out as reported bugs from the customer.

Figure 1. Defect injection reason count and percentage.
Defects related to poor test cases are excluded from this summary.

Figure 2 summarizes the information in figure 1 into two basic categories. It turns out that 55% of reported non-test defects resulted from requirements or communication issues. While 18% of reported non-test defects were injected directly by the development team. This 18% includes oversights (things you just forgot), education (things you should have known how to do), and mistakes (things you understood and meant to do but botched somehow). We also included process issues in which the process we were following created an opportunity for a defect that was actually injected into the software and escaped to the customer for review.

Figure 2. Defect injection reason count and percentage, identical to Figure 1.
Shows that communication and requirements issues account for 55% of all non-test
related defects. Further, 18 % of defects were injected directly by the team and might
be avoided through better training, testing, or peer reviews.

While understanding why we injected defects is interesting, understanding where the defects are injected is important to creating a plan for doing something about it. Figure 3 shows the defect count and percentage for each of the main areas of the software product we were developing. The top three trouble spots, our "bug farms" were the display configuration (user interface), test cases, and data conversion and extraction code. That poor test cases were one of our greatest sources of defects is extremely disturbing.

Figure 3. Count and percentage of defects injected into different software modules.
This graph includes defects related to tests.

Putting it all together (figure 4) really shows our trouble spots and gives a pretty fair view of where the team needs to improve. Also important, the data shows low value areas and establishes a baseline for other modules and reasons that we don't need to focus on currently but also don't want to see suddenly increase over time.

Figure 4. Count of defects by module by reason injected.

When I look at this data, I see that communication in general is a pretty big problem. There were acute problems when trying to communicate requirements in the UI, especially late feature requests and changes. Finally, my team just couldn't get it together for testing and made mistakes all over the place.

In the remainder of this post I am going to focus on how we dealt with changing requirements and how we might improve. Testing is hopefully a topic for another day...

Change, Essence, and Accident in Agile Software Development

As you can see, we had a lot of defects escape implementation to customer review. Digging a little deeper, close to 50% of our defects reported coming out of customer review had to do with UI issues. Digging deeper still, the majority of these UI issues weren’t bugs but rather changes to the originally specified requirements. In other words, we built something that we thought was correct, but after the fully implemented feature was reviewed the customer decided to change how the feature works.

This is rework. Changes like this are waste.

"But wait!" you say, "You are an agile team! You should be embracing change! Change isn’t waste, change is the name of the game!"

Of course we embrace change. Every UI issue reported by our customer was a learning opportunity and a chance for us to make better software for them. But that does not change the fact that change has a cost.

With any software development there is going to be change. Agile methods anticipate and even embrace this uncertainty. That’s the whole point. The question we should be asking is this. How much change is expected? And what is a reasonable cost of change in a project?

In other words, how much rework is essential to an Agile project and how much is avoidable, accidental rework?

Regardless of whether you are embracing change or not, reducing rework is still a good thing. Getting it right the first time, before you have implemented a feature and delivered it to the customer for review is the most cost effective way to build a system.

But at the same time an essential factor for success is to create feedback loops for learning more about your users and what they need. Change is waste. Change is also necessary.

Applying Knowledge from Defect Classification

In our case, after looking at the data and reflection on what we saw the team felt strongly that there is room for improving our requirements process. We had a hunch that requirements were a problem but we had no idea that it was as bad as it was. The next problem we need to solve was what to do about it.

Looking back at past projects, it seemed like requirements churn was a problem but it was never as obvious as it is now. We've tried several different approaches over the past year and a half with mixed results.

"Shall statements" didn't work because they aren't descriptive enough. The traditional method for specifying these requirements are in an Excel spreadsheet. This encourages brevity and makes for bad requirements.
User stories are slightly better but the majority of our customers are not familiar enough with the format. The end result is a confused customer and poorly specified and validated system.
Letting the customer drive the requirements has also been mixed. The customer is usually very happy with the format (they choose it) and eager to review. Unfortunately, the requirements are usually lacking important information or the customer's chosen format is otherwise awkward for the implementation team. In other words, it's good for them but not good for us or the project on the whole.
Use cases were extremely successful with the development team, but we experienced similar issues as user stories when presenting the use cases to the customer. The format and information was just abstract or foreign enough to make the representation a challenge.

For a very long time I had assumed that the problem was with the requirements representation itself. I assumed that there must be some "holy grail" format for specifying functional requirements. After examining this data and reflecting on our past experiences, I'm not so sure any more. User stories have been extremely successful within other parts of our engineering organization. And I have firsthand success stories applying use cases.

Knowing that the majority of our requirements issues stem from change is the enlightening nugget of knowledge here. Changes are a positive thing. But can we flush out these changes before we fully implement the feature?

Based on our data, here are some things we are thinking about trying on our next project.

Apply use cases and take more time upfront to educate the customer on how to read and evaluate them.
Try to match the right requirements specification type with the customer's behavioral and technical inclinations.
Try out a different requirement specification representation. We're working on a new idea, "FAQ Requirements." Maybe an easier to read and review representation will resolve some of these problems?
Wait to conduct a UI workshop until after at least part of the system has been implemented. Our hope is that by this point the customer will have seen some aspects of the system and we will have had an opportunity to teach them core concepts in the application space.
Spend more time prototyping using cheap prototyping methods. In some cases, paper prototyping might have resolved some of our uncertainty much more quickly than a fully or even partially implemented feature.

The goal in making a process change is not to remove the feedback loops or eliminate change by trying to get it right the first time. Instead the goal is to help us fail faster, cheaper, earlier, and more often. We want to make the mistakes early, before they are realized in code, when change is cheap. It's a lot easier to change a few lines on a whiteboard than it is to rewrite an entire code model.

Failure is a mechanism for learning. This education needs to be as cheap as possible. Our focus over the next few months is going to be figuring out how to make our requirements representation so good that it fails fast as a requirement. Hopefully this way we can avoid waste from rework.

Dealing with Constraints in Software Architecture Design

2014-10-22T08:30:00.000-04:00

Definition of constraint (n)

Bing Dictionary

con·straint
[ kən stráynt ]

limiting factor: something that limits freedom of action
state of restriction: a state in which freedom of action is severely restricted
lack of spontaneity: a lack of warmth and spontaneity in somebody's manner or in the atmosphere on a particular occasion

In software architecture design, constraints come in two basic flavors - technical and business. On most projects there are only a handful of constraints, but these constraints are a highly influential architectural driver. Constraints, as the dictionary definition above indicates, are a limiting factor and severely restrict options for making design decisions. They are also fixed design decisions that cannot be changed and must be satisfied. You could think of constraints as the ultimate non-negotiable, "must have" requirement. It is for this reason that architecture design constraints must be created and accepted with care.

Anthony Lattanze of Carnegie Mellon University, author of the Architecture Centric Design Methodology and one of the ATAM creators, likens design constraints in software architecture to "load bearing walls" employed in traditional building architecture. Load bearing walls of a building are extremely strategic and strongly influence the shape and form of the building. What's more, once these essential walls are placed and bearing weight it becomes cost prohibitive to move them.

Technical and business constraints are similarly load bearing. Many design decisions will hang on fixed constraints, further cementing them into your system's design. Adding a component or even refactoring architectural styles might be costly but it can still be done. Deciding to jump from .Net to the JVM requires demolition and rebuilding.

As architecture design constraints are so important it's worth taking some time to understand them in greater detail so you can properly deal with them when they arise.

Technical Constraints in Software Architecture

Technical constraints are fixed technical design decisions that absolutely cannot be changed. Most often technical constraints are provided by stakeholders (perhaps after some digging) at the outset of the project. Sometimes a team may choose to create a constraint, perhaps to simplify the world in which they are working. Regardless of the origin, these architecture drivers are technical in nature and are considered unchangeable and indeed very soon become so.

There are many common examples of technical constraints that you've likely seen.

Programming language - often times a specific programming language will be required for various reasons. For example, the customer may be a Java or Ruby or Microsoft shop. You might simply prefer a certain language over another, or have specific expertise that dictates using a particular programming language. Nearly always, once you've picked a language you are stuck with that choice for the remainder of the project,
Operating system or platforms supported - It must work on Windows, or Linux, or iOS, or Qt on Solaris, or IE 6 on Windows XP, or ... building software that does not satisfy the platform constraint means you have failed to design a software system that satisfies stakeholders' key concerns.
Use of a specific library or framework - Sometimes a specific library might be required to be used. The specific origin might come from the business but the influence is very technical. A common example at many companies is the use of specific open source libraries. Some companies might require that open source always be used. Other companies, might have an approved list indicating which open source software may be used. An interesting example at IBM is that we are required to target the "Blue" JVM, IBM's JDK for all JVM-based projects.

Business Constraints

Business constraints are unchangeable business decisions that in some way restrict the software architecture design. Business constraints are similar to technical constraints in that they are decisions that cannot be changed, but rather than influencing structures directly through technology, the influence occurs indirectly through business decisions. It might be easiest to discuss some examples.

Schedule - "We have to be ready to demo at the trade-show in November." The final delivery date is fixed based on input from the business and cannot be changed. It is now up to you to design a system that can be built within the required schedule.
Budget - I've found that this is less true for product development, but nearly every development contractor must learn to deal with budget constraints. In some cases you might be able to increase the budget, but often budgets, whether fixed through contract or expectations, become constraints on the project and team building the software.
Team composition and make-up - In some cases there may be requirements that specific personnel be used or not used during project. For example, someone may be unavailable, already committed to another project or you may be required to include specific individuals, perhaps for training purposes.
Software licensing restrictions or requirements - Often times you will be required to use specific software components because they are already owned. I've also seen situations where components have been used in very... interesting... ways to satisfy specific licensing restrictions. Legal considerations are extremely constraining.

Identifying Constraints

Constraints originate from a variety of sources. In my experience, more often than not at least one stakeholder has a specific concern that will result in a constraint. For example, the business line or some other stakeholder might decide that IE6 running on Windows XP must be supported (and in spite of our resistance it remains a requirement). If you've ever created a web application for a federal agency of the US Government then you'd know that this was a constraint on all web projects until very recently. United States law requires support of a specific operating system and browser. Period. That changes when Congress passes a new law.

Constraints can also be created before a project even starts, during procurement. In my previous role at IBM as an application engineer / consultant with expertise in the Watson Explorer platform (a/k/a Vivisimo Velocity or Data Explorer), every project on which I worked was constrained to use the Watson Explorer platform. As a consultant I was only brought in once the decision had been made to purchase and deploy Watson Explorer. This constraint is fixed during the pre-sales and sales phase of the system.

It's common that quality attributes and functional requirements will be offered up as constraints. It is a huge mistake to treat a quality attribute as a constraint, even one that is extremely high priority. Quality attributes are NEVER constraints.

When starting a project I advise teams and teammates to be very conservative when fixing constraints. Challenge constraints. Do not accept them lightly. Is this requirement an absolute must-have or is it just extremely important? Is the decision a strong suggestion or a threshold of success for the system?

Over time, design decisions you've made can become constraining but it's important to distinguish between the constraints you are given and the constraints you've created for yourself. Ultimately, the constraints you give yourself can change if you need them to while constraints you are given are non-negotiable. This is why constraints should be considered a distinct architectural driver and tracked explicitly from quality attributes, functional requirements, and specific design decisions.

Architectural Drivers: Building Blocks for Decision Making

2014-10-15T08:30:00.000-04:00

Architectural drivers are formally defined as the set of requirements that have significant influence over your architecture. In other words, there are some requirements that will help you decide which structures to pick for your system design and others that are less consequential in the context of software architecture. This implies that it is extremely important to get the architectural drivers right early in a project, as architecture structures will become more difficult to change as the developed software becomes more realized.

Thinking about early requirements as "drivers" can also help you to create traceability from stakeholder concerns to specific architectural decisions made. Having this traceability is great because it helps to promote important qualities in your team, such as agility. With traceability you can make changes on purpose and with confidence as you understand the genesis for structural decisions.

What are Architectural Drivers?

Architectural drivers have significant influence over the design of a system and are the determining factor over whether one design is “better” or “worse” than another that produces the same features. Architectural drivers are nearly always divided into four categories. Some people might go with only three categories (talking generally about constraints), and others might include other things as drivers, but if this is your first time working with architectural drivers you should start with these four categories before tailoring.

Architectural drivers can be one of the following.

Technical Constraints
Business Constraints
Quality Attributes
High-Level Functional Requirements

Let's go into more detail for each of these categories.

Technical constraints are technical design decisions which absolutely must be satisfied in the architecture. These are generally fixed from the beginning of a project and may be expensive to change as the project progresses. Constraints, as the the name suggests, cannot be changed.

Business constraints are decisions imposed by business considerations that must be satisfied in the architecture. Like technical constraints, business constraints are generally fixed from the beginning and have a significant influence on decision making for the design. Likewise once in place these are considered unchangeable. An example might be a specific delivery date, budget, or

Quality attributes, sometimes called quality requirements, are specific criteria by which a system’s operations can be judged – that is, how well a system performs specific functions. Quality attributes greatly influence the design of the system. In many cases, the same functionality can be achieved in a number of ways and so it is the quality attributes that ultimately determines when one design is better or worse than another. Quality attributes receive a significant and thorough treatment in Software Architecture in Practice and you can read a sample chapter introducing quality attributes online. Quality attributes are recorded as scenarios that describes a qualitative response within the context of specific system functionality. For example, performance is a common quality attribute, but performance when the system is doing what?

High-level functional requirements provide an overview for what the system will do, the functions and features of the system. These might take shape in any number of formats from epics to stories to use cases to ye olde "shall" statements. While quality attributes might determine specific structures you choose, the ultimate goal of the software you are building is to build achieve some bit of functionality. From an architecture perspective, you won't go to exhaustive detail, and indeed it's best not to go into too much detail as diving deep too early can seriously delay the project.

What about Non-Functional Requirements?

If you've read any of the software engineering literature, you'll see the term "non-functional" requirements used fairly often. The basic thinking was that there are these things that are functional requirements, and then there's things that are not functional requirements, non-functional requirements. Setting aside the fact that calling requirements "non-functional" is kind of silly and certainly not precise, this term comes up enough that it's important to understand it in the context of architectural drivers.

Most of the time people are referring to constraints and quality attributes when they talk about non-functional requirements. Unfortunately, most of the time teams will lump all "non-functionals" into a single list. This is unfortunate because knowing specifically what kinds of drivers you are dealing with gives you a lot of decision making power.

Functional requirements are negotiable and must be shown to be correct.

Constraints cannot be changed once fixed.

Quality attributes imply trade-offs and often can't be directly tested until very late in the system's lifecycle. As a result you will need to find proxies, run experiments, and reason about the structures you've selected to determine whether or not you've promoted the right quality attributes.

Five Challenges for a Maturing Discipline

2014-05-20T08:30:00.000-04:00

[At the opening for SATURN 2014 I presented five challenges for the software architecture community as a means of kick starting our brains for the conference, but also a serious reflection on the state of software architecture practice. My hope is that these challenges are something we can work on together as a community and are my attempt at capturing something actionable that we can do to accelerate software architecture's maturation. On the surface these five challenges are pretty simple. Of course if it truly were simple then it wouldn't be much of a challenge. Think about them. Give some of them a try. Let me know what you think.]

A little less than 20 years ago, Mary Shaw and David Garlan published a book reflecting on trends they noticed were emerging in software development. The book's subtitle was extremely fitting: Perspectives on an Emerging Discipline. My reaction to the book, having read it for the first time for a software architecture class in 2008, was that it was grand in scope yet unassuming. Having had the opportunity to meet and work with both David and Mary while a student at Carnegie Mellon, I found that the tone of this landmark book is more a reflection of who they are than anything else. Essentially they proposed to the software engineering community, "Here are some interesting ideas and trends we've noticed and we think they are worth taking seriously, but these ideas need some further thought."

While Perspectives was certainly intended to be a unifying, landmark work, Mary and David had the wisdom to recognize that this was only the beginning, that this emerging discipline of software architecture must continue to mature, and that others must be responsible for the growth of the practice.

Fast forward to today, and the success of conferences such as SATURN show that software architecture as a discipline is maturing, that we are continuing to improve the state of practice with every new software system shipped. But while we are learning more and getting better and better as a community every year, it is also becoming apparent that we still have a long way to go.

Software architects are expected to be a very different kind of leader than what is commonly required of folks in a leadership role. Software architects need to maintain a focus on business, technology, and people simultaneously. Software architects are expected to be experts at the largest granularities of abstraction while at the same time fully comprehending the implications of the tiniest technical details.

This is a problem for us because mastering both of these worlds – coding in the large and coding in the small – is extremely difficult. In the past decade or so we've become really good at coding in the small as evident in the thousands of flowers blooming on Github and other code depots. As an industry though we also seem to lack a general self-awareness and discipline when it comes to understanding why we make the decisions we make. To me this seems like a problem in how we educate software architects.

While it's great that we can force success through brute force and raw smarts, if we as an industry ever hope to reach a point where someone can master software design basics before becoming a grey haired master, we need to continue working to mature our discipline. Currently it takes 10 - 15 years of experience for someone to really master software design. The prevailing wisdom is that a person is not even ready to learn software architecture - coding in the large - until they've practiced software development for at least 2 – 5 years - coding in the small. At the same time, elementary school students are practicing geometry, calculus, and trigonometry -- mathematics that took some of the greatest minds in the history of the world years to figure out. Certainly we can do better with software architecture than we are currently doing. I'm not suggesting that education is a substitute for experience. I am suggesting that we can improve our software architecture education and make software architecture more accessible and easier to learn.

As an emerging discipline, that we have problems like this is actually a good thing. Software architecture is growing up. The discipline has emerged and as a discipline software architecture still has a lot of growing and maturing to do. My hope is that these challenges will not only get you thinking about where we are going with software architecture, but also that you will accept some of these challenges and at the same time help continue to improve the maturity of the software architecture discipline.

Five Challenges for the Maturing Discipline of Software Architecture

Increase diversity – technology, age, gender, cultures, everything. Having diverse perspectives on a team is a force multiplier for success. Every team should strive to have rich diversity. Men and women. Young people and older people. Different cultures and backgrounds. Different universities. And perhaps most importantly, different technical and work experiences.

Ecologists often refer to biodiversity as one of the most important traits for determining the success or failure of a species. Drawing an allusion to evolution is not an accident here. As your system evolves, the more diversity that is available in the ecosystem the better fitness that system is likely to have. Learn a new programming language. Learn a new framework. Every once in a while, get out of your comfort zone and seek out people and communities with opinions or practices different from your own. You will learn something new and the results will be surprising and good.
Write more code. There is this nasty rumor that software architects don't write code. Plain and simple, anyone who calls themselves an architect without dirty hands is not an architect. I know many agree with this, but to overcome this negative perception we need to write even more code.

I am not suggesting that we should write more software, but rather software architects need to remember their roots. Writing more code does not mean you should be writing production code – in fact it's probably best that some (most?) team leads and architects stay off the critical path for a project. Non-production code is still extremely important. Direct observation through experiments is one of the most powerful tools we have in software architecture. Learn by doing.
Put humans first. As software architects we are very much focused on software systems. So much so that it seems we sometimes forget that the whole point of the software and the documentation and the models that we build is so we can provide value to people. Human centered design is an extremely important trend in software development right now and the software architecture community is completely missing out.

Who are the humans software architects care about? Downstream programmers, maintainers, testers, customers, managers, fellow teammates – the user too, but as software architects we are responsible for the experience of everyone who interacts with the systems we build, not just the users. Put humans first in the documents you write and the systems you design. Empathize with the developers, the testers, the managers, the integrators, and not only the users.
Create a sense of craftsmanship. We've all seen it. That diagram that was created just so someone could check off a box on a process. The font is too tiny to read. There are innumerable lines and boxes. Nobody really uses the diagram because nobody wants to take the time to understand it. The person who created a diagram like this straight up did not give a shit. Design is as much about results and process as it is about aesthetic.

Part of putting humans first is caring about what you build. Embracing craftsmanship means only creating software, models, documentation, plans, and so on that you would be proud of. Craftsmen strive to produce work that they would proudly declare as theirs, and would be willing to stake their reputation on what they produce.
Share what you know. As software architects, our job is just as much about teaching as it is about programming and technology. A software architect who can't share their ideas is neither a useful leader nor an effective designer. Conferences such as SATURN are critical to creating great software architects and offering platforms for sharing. So too are less formal self-publishing platforms such as blogs, YouTube, LeanPub, Hangouts, Stack Exchange and so many others. Collaborate, share, teach others – this is the only way software architecture can continue to grow and mature.

These are the challenges of a maturing discipline. These are my challenges to the software architecture community. Are you up to the task?

I'm going to SATURN 2014 and you should too!

2014-03-23T17:55:00.000-04:00

The SATURN 2014 conference is taking place this May 5-9 in Portland, OR. You should consider joining me at this essential software architecture conference. Register now online while tickets are available!

After graduating from Carnegie Mellon with a Master in Software Engineering degree, I needed a way to stay relevant on software engineering topics. As great as it would be, taking time off of work to earn a new degree every 5 to 10 years just doesn't work. It’s time consuming and stressful, not to mention cost-prohibitive. Rather than going back for more degrees, I decided that I would attend at least one professional conference every year. Ideally, I also try to present or write at least one paper every year too, but attending at least one conference is the most important goal.

This year my conference of choice is SATURN 2014. I decided to attend SATURN this year for a number of reasons.

SATURN is the largest conference dedicated exclusively to software architecture practitioners (not academics) in North America. I will only attend conferences where I can learn something that I can actually use in my job. My expectation is for a high density of practical knowledge.
SATURN is one of the few conferences in the world dedicated to software design principles and practices. There are lots of conferences out there on methods. Tons of conferences on programming and specific technologies. SATURN is one of the few conferences that examines these topics from the perspective of architecture and software design. This perspective if desperately needed sorely lacking in software professionals today.
SATURN is an established conference. The conference is co-hosted by the Software Engineering Institute and IEEE Software, and this year marks the 10th consecutive conference held. The SEI and IEEE are responsible for having established and promoted nearly every standard practice in software architecture that has ever existed. Their participation in this conference is a strong indicator of the kinds of topics that will be covered.
SATURN is attended by the folks who define software architecture. Here are just some of the software architecture thought leaders who are speaking at SATURN -- Philippe Kruchten, Rick Kazman, Len Bass, Robert Nord , Felix Bachman, George Fairbanks, Ipek Ozkaya, Jeromy Carriere, Rebecca Wirfs-Brock, Ted Neward, Linda Northrop, and more…

If you measure a conference by the number of speakers who have a Wikipedia page, this conference has at least 5. Bonus points if you can find them at the conference.
SATURN has a good mix of presentations, workshops, and longer tutorials. The tutorials and classes are top-notch. I personally am looking for advanced tutorials and workshops to attend and SATURN has these. There’s also a couple with introductory information for software engineers just getting started in the world of software architecture.
SATURN is hosting a co-located Open Space facilitated by Diana Larsen. I’m an explorer. When choosing a conference I look for opportunities to explore beyond the established program. I love a good Open Space conference and I am very much looking forward to having an Open Space filled with software architects. I am even more excited that Diana Larsen will be the facilitator. She has done an amazing job facilitating at the XP conference in Europe every time I have been.
SATURN keynotes this year look amazing. It’s unusual for me to attend a conference based on keynote speakers. Most of the time I’m not that excited about the big speakers and I’m really looking at the overall topics in the conference. That said, the SATURN 2014 keynote lineup looks really solid.
SATURN is held in Portland, OR this May 5-9. Timing and location also play a significant role in my deciding what conference to attend. I have always wanted to visit the Pacific Northwest and Portland in particular. I’ve heard so many amazing things about Portland that I really need to see it for myself. May is also a great time for a conference, at least for me. I must admit that if I could afford to go to Europe this year, I am extremely tempted by XP2014 in Italy.

I should mention that I am serving as the technical chair for this year’s SATURN conference. You should view my serving as the technical chair for SATURN as the greatest possible endorsement I can give a conference. Given the massive time commitment required to plan and organize a conference, I can’t afford to spend time on something that I do not feel is massively important to software architecture, and something that is not great. SATURN is important to our profession. SATURN is a great conference. I very much hope to see you there this May.

Other Conferences of Note:

Software Architect London -- I would call this SATURN's sister conference for Europe. I have not attended, but it is pretty close in terms of quality of program to SATURN and a good choice if you are from Europe.
XP 2014 -- outside of SATURN XP is my favorite conference after having attended twice. The biggest problem for me is that it is in Europe. Like most Agile conferences it is usually light on software architecture (albeit the strongest architecture presence of the Agile conferences I've attended), but the locations, excellent planning, and dedicated research track more than make up for this.
Agile 2014 -- Agile has been expanding to include Agile coaches and UX topics, which has shifted the focus away from core software development. That said, if you pick the right tracks this is a great conference.

Formal Models: I Hate You, I Love You

2014-01-02T07:00:00.000-05:00

My view on formal methods is somewhat... nuanced. In spite of this I'll go ahead and start with a ridiculously strong and outrageous assertion that I can use as a toe hold for further discussion: Formal specifications do not have a place in agile software development. Don't get me wrong. As a software engineer I like the idea of formal specifications very much. They are precise. They are unambiguous. They are... engineering-y. But as a post-modernist software engineer I also think formal specifications are kind of a dumb idea. They are tedious. They are cryptic. They are... anti-human.

For decades software engineers seem to have put formal specifications on a pedestal as if they were the Michelangelo's David of requirements. But while Michelangelo's sculpture of David manages to perfectly capture the essence of humanity in marble, even the most elegant formal specifications fail to fully capture the perfect essence of desired functional behavior of a software system. The fatal flaw of formal specification comes from removing the human element and assuming that cold mathematical precision is an adequate model for communicating and comprehending human desires. (Yes, I am creating a straw man, but please allow me to beat him up just a little more before providing a counterpoint.)

A common example formalists use to demonstrate the strengths of formal specification (and weaknesses of plain old English prose) comes from the alleged ambiguity and confusion created by various warning signs posted by escalators throughout the UK. "Dogs must be carried," the signs read. "What if I don't have a dog? Can I still use this escalator?" the formalists rhetorically ask, I always imagine with a smug grin.

The more important question:
What happened to that dog's butt?

This argument of course is completely absurd. Contextually, no confusion exists. If you have a dog, then carry it up the escalator. The formalist may ask next, "What if I only own a cat?" Ugh. Formal models may be more precise but this does not mean they are necessarily more correct or more complete. Whether using natural language or a formal model, the question of cats (or goats or fish or pythons) is still a good one. Chances are good you may ride the escalator if you are not carrying a dog. Also, you probably should carry your cat too. (Another great example for those interested: the TSA wants everyone to pack gel-filled brassieres.)

The common fallacy is that formal specification seems to encourage the incorrect assumption that more formal a model the more correct it is. This confusion is likely created by confounding precision with correctness. The goal is to create the most correct and precise model as possible so that you have an irrefutable representation of the system, a representation that has no room for argument or confusion. This is certainly a positive intention, albeit slightly misguided. From an Agile perspective the goal should not be to create something that cannot be refuted but rather to facilitate effective communication. In a world where collaboration is strongly preferred over negotiation perfection is the enemy that will drive you and your customer apart. There is no thin red line in a healthy customer-team relationship.

This is not to say that ambiguous requirements are desirable. The confusion is in the message sent by the type of specification that you use. The mathematical obfuscation and high-brow attitude of formal modeling sends all the wrong signals and can even make it more difficult to create a correct model. For the average customer, offering up formal specifications as an irrefutable, unambiguous, no room for error model is the equivalent of saying, "Hey client, I don't care if you can't understand this and your needs are not important to me. I really just want to cover my ass here so I can throw your desires over the wall for construction." Precision, correctness, completeness, and a lack of ambiguity are important traits for any specification, but so too is effective communication and a certain degree of flexibility to forge a new path as new information becomes available.

Formal specifications seem to be born from a fixed mindset, a good match for folks who cannot (or simply refuse to) operate without exacting precision and firm, non-negotiable interpretations of "right" and "wrong." Unfortunately, "right" and "wrong" are very relative notions on most software projects and "better" or "worse" is the currency in which we deal. Understanding whether something is better, or even just good enough, is often relative and can only be fully understood once you have seen a solution if only a partially working one.

And yet...

And yet, I am still an engineer at heart and do not want to throw the formal baby out with the bathwater. Formal specification might still be useful in an agile context as long as the methods complement agile values. Behavior Driven Development, design-by-contract with predicates and post-conditions, automated functional testing – these are all models in the spirit of formal specification without the high-brow, non-negotiable attitude. What's more methods like these aim to enhance communication between customer and team, and even within a team, rather than obfuscate it. A good old fashioned state machine sometimes really is the best medicine, but in my experience it is a tool that can only be used effectively with the right set of stakeholders.

Communication is something everyone can agree is a good thing, at least in theory. Even the most ardent Z supporters will tell you that one of most valuable aspects of a Z specification is in the prose that accompanies the model. Meanwhile, those same Z supporters will breathlessly defend formal methods, even though they just admitted that the informal part is one of the most valuable pieces, that the formal model really should not exist without the explanatory prose. The model is really just a tool, a means to an end for achieving clarity in thinking.

What I am proposing is not to abandon formal specifications. Formal modeling is an awesome tool - when used with the right frame of mind. Formal models that embrace agile values and promote humanist qualities are the ones you should be using. Why not make that model, or the prose surrounding it, executable? Why not rely on contextual or cultural cues to enhance meaning? Why not use comprehensible prose, diagrams, sketches, prototypes, tests, and working software in combination to reduce ambiguity, enhance precision, and improve correctness while enabling true collaboration?

Pragmatic, human-centered formal methods have an important place in Agile software development and should be embraced wholeheartedly.

Rapid Software Architecture Exploration

2013-06-24T09:00:00.000-04:00

At XP2013 this June in Vienna, Austria I ran a brand new workshop titled "Rapid Software Architecture Exploration." In the workshop I presented a framework for thinking about software architecture design based on my essay, Exploring the Design Space, and gave participants a chance to put the theory to practice using four different exploration activities. Slides and handouts from this session are available online.

Huge thanks to everyone who came to the workshop. Everyone did an amazing job and really ran with the activities we did. I think the end results were very good -- surprisingly an almost fully architected system in only 45 minutes (plus some of the prep work that I did ahead of time). I have no doubt, now even more so, that with a good facilitator and the right people in the room you can accomplish in hours or a few days what traditionally could take weeks.

The session was great because you guys made it great. Thank you! If you use these workshops with your teams or customers, let me know how it goes, I'd love to hear from you!

System Properties Web

The system properties web is intended to be a light version of the traditional Quality Attributes Workshop. Combining structured brainstorming or a quality attributes taxonomy questionnaire with a spider diagram for capturing concerns, stakeholders can quickly identify raw quality attribute scenarios and create a visualization depicting the "signature" of the system.

A group brainstorms raw scenarios and concerns.

Categorizing concerns helps create a visualization to assist in the workshop.

The stakeholders in this group seem most concerned with reliability and availability

While this stakeholder group identified concerns across many quality attributes,
notice that neither of the examples identified many scalability concerns.

Round Robin Design

Round Robin Design is a method for quickly generating many ideas and then helping to converge those ideas. The idea is simple. Working alone or in small groups you design a view of the system and then pass your design to another group for critique and yet another group for improvement. Constraining the time forces you to get creative and actually design something while sharing your designs exposes you to new ideas and creates a sense of shared ownership among the team.

Everyone got very quiet during this activity... Deep in concentration.

Another group of architect's working quickly to
design a physical perspective of the system.

Risk Storming

Risk Storming is a workshop proposed by Simon Brown of "Coding the Architecture". The basic idea is to use diagrams of the system as a kick start for brainstorming risks in the current system design. These risks then become the basis for designing experiments, research, or further detailing design for specific parts of the system.

Working together as a group, both engineering risk and
project management risk should be considered.

Each sticky represents a risk or worry. Place the stickies directly on the diagram
in the area to which the risk applies.

Using different colored stickies also helps create a visualization of risk
in the system - or at least show what folks are worried about.

At least one view of the dynamic, physical, and static
perspectives should be evaluated. Context diagram too if time is available.

Stakeholder Mapping

Stakeholder mapping is an exercise introduced to me by Ari Font. This 30-45 minute activity is a great way to start a full day of workshops. I like this activity a lot. It is always a surprise to see how many stakeholders and stakeholder groups actually have influence or could be influenced by the system that you are building. This is also an activity that helps break the ice and sets the tone for group participation.

Collaborating on a stakeholder map for their group.

One of the group's maps.

Groups only had about 10 minutes and quickly identified
more than just the usual suspects of stakeholders.

Reflection

Overall I felt this workshop went really well. Front loading the theory and presenting the workshop activities first gave folks who were not interested in hands on practice a chance to learn something without committing to a group learning activity. I understand it -- after a few days at a conference, a sit and listen lecture can be just what you need sometimes. For those who stayed, the hand on parts were a fun and effective learning experience.

I was extremely impressed with how quickly everyone picked up the practices and applied the theory. I take this as a good sign that the framework for thinking about design is a good approach. I know that I've been successful using this approach and these practices, but a true test is when someone else is able to apply the ideas on their own. A future variant of this workshop might be even more aggressive in how much the teams design on their own.

XP2013 is one of my favorite conferences. It's got a great mix of researchers and practioners, and a workshop like "Rapid Software Architecture Exploration" really benefits from this mix.

Call it what you will -- rapid exploration, lean software architecture -- combing best of breed practices from other design disciplines with lean versions of common architecture workshops seems like a winning combination.

Overcoming Team Dysfunctions: From Storming to Awesome

2013-05-06T09:00:00.000-04:00

Forming, storming, norming, performing, mourning -- the stages lof team formation. I’ve experienced this natural progression a few times, transitioning from a group of strangers to a well-oiled machine – to friends.

One team I worked on figured out a way to collect data and measure our journey from new to awesome. We didn’t know about this standard team lifecycle when we started, but that didn’t mean we could avoid it. In fact, as you’ll see in this essay, embracing the steps of team formation helped us move through the phases and become a highly efficient team that much faster.

Forming and Storming

We were a team of five working in an industrial-like university setting. Our mission was to build a new web-based workflow tool in support of the security requirements elicitation methodology, SQUARE. We had a budget of about 3,800 engineering hours and a delivery date in 16 months.

Overall my team had worked together well from day one, but coming together as a team was not easy. Our biggest challenge involved overcoming the inherent cultural biases and assumptions that everyone brings with them wherever they go. This was initially amplified for us being from so many different places. For the first month we fumbled because each of us simply assumed that everyone else worked the same ways, that we had the same values. Of course we did not and this realization created riffs of distrust within the team.

To consider us a team at all in the beginning was really absurd. We were from all over the world and everyone was new to Pittsburgh. At the time this was one of the most diverse groups of people with which I had ever worked with teammates hailing from the US, Panama, India, Bangladesh, and Taiwan. In my case (and I’m sure it was the same with everyone) I came from a team that worked like a well-oiled machine. The team I worked with before had been together for over a year and had been working efficiently for over half that time.

Leaving an amazing team and going to a new one with no trust was jarring and excruciatingly painful. After the first month this new team could barely work together on anything. We were all still in mourning after having left our previous, awesome teams. And things were not going well as we were getting to know each other while at the same time trying to get work done.

Conflicts, distrust, missing commitments – we were a full on storming team. And we seemed to be stuck there.

In hindsight, the biggest problem we faced in the beginning was that we didn’t know we were a dysfunctional team.

Identifying Team Dysfunctions

One tool that helped turn everything around for us was the Team Dysfunctions Survey. The survey comes from Overcoming the Five Dysfunctions of a Team by Patrick Lencioni. The book is a field guide for, as the title suggests, overcoming the various dysfunctions teams face. The book consists of equal parts theory and application. Each chapter talks a little about a dysfunction and then presents activities that can be used to help overcome the problem. The material will look familiar to anyone who has ever attended a leadership retreat but it’s presented in a disciplined, well organized way so that any team leader can use it to help bring the team together.

The Team Dysfunctions Survey is a brief 15 question quiz that all team members take to gain a quick assessment of how everyone thinks the team is doing. When my team took it we kept the lowest score in the group from each of the five sections of the quiz. Our thought was that if one person on the team feels there is a problem in an area, then the whole team has a problem. So if one person felt a lack of trust, for example, then trust is an area in which the entire team would need to work on further.

The quiz quickly revealed some important aspects about our team. One of the most important things I learned was that I, as the team leader at the time, had a skewed sense of how the team was actually coming together. With data from the survey in hand and a way of thinking about building healthy teams I had the power I needed to pinpoint the exact areas we needed to work so we could come together as a team.

Lencioni describes the five dysfunctions as a pyramid where the things at the base must be present for the things at the pinnacle to be successful. For example, trust, at the base of the pyramid, is the most essential part of any team. Not surprisingly, this was also the area where my team was having the most trouble as revealed by the Team Dysfunctions Survey. As such, building trust within the team became our top priority.

Building a Foundation Based on Trust – Getting Past Storming

Really the only way that you can trust someone is if you know them on a human level. The only way to know someone is to share something about yourself with them. This sharing involves creating a level of vulnerability and building a bond of trust based on that shared experience. Without this trust, you will not work as well together and this will negatively impact the quality of work the team is able to produce.

The easiest way to get to know someone is pretty obvious since we do it almost every day: hang out. To kick off our trust building effort I hosted a dinner party at my house where we ate lasagna and played Wii Sports for a few hours. If I recall correctly, I think we even buzzed one of fellow our teammates hair! Letting everyone see where I lived and getting everyone together in a relaxed, non-work setting made it easier for folks to open up and tell stories about where they came from, what they valued, and how they thought about the world.

Hanging out was really good for us, but we had a lot of work to get done and not a lot of time to do it. To accelerate trust building we played some simple icebreaker games. One example was the M&M game. In the M&M game, each team member gets a small pile of M&Ms and everyone sits together around a table. Going around the table, one at a time, each person picks up an M&M and tells the group something about themselves and then eats the candy. The color of the M&M determines what the subject will be, red for family, yellow for job, green for hobbies/personal interests, blue for childhood, orange for undergraduate, and brown for future. In the ideal situation, each person chooses a wide range of colors and talks for less than two minutes at a time for each story they tell. This game was simple and effective for us.

Measuring our Dysfunctions Progress – Norming and Performing

So did it work? Another interesting thing my team did was to continue using Lencioni’s survey as a means for assessing how we were doing. To keep things honest, there was a 60 second time limit on filling out the survey and the results were collected anonymously. We charted our progress over time in each of the five areas. Keep in mind that the things at the top of the pyramid probably aren’t valid unless the things underneath are solid. In this case, solid is a score of 6 or higher on the survey.

My team’s results over the course of the project are charted in the graph below. The score for each dimension is shown over time. Lencioni’s terminology is strange to use in this way, so it might be easier to negate the ideas in your head when reading the graph – for example a high score for “Absence of Trust” really means “We have Trust” while a low score means there is no trust among the team.

A few interesting points from the graph above:

The surveys were not taken at regular intervals. Because of the way the capstone project is designed, we only spent 12 hours per week during the Inception and Elaboration phases and 48 hours per week during construction. Time between surveys was probably 1-2 months on average (just enough time to forget what the questions mean).
The Inception postmortem was the end of the fall semester. We were feeling pretty good, having survived our first semester.
Elaboration iteration 3 was the end of the spring semester. We had just received bad marks on the architecture we had spent all semester designing and realized we had more work to do. Notice that trust is at an all-time low and conflict is at an all-time high. This means that issues were being flushed out but it wasn’t always in a healthy way. In other words, there was some “blame game” going on.
We focused on trust again during the summer Construction phase with release parties at the end of iterations and by using a daily team stand-up meeting to keep everyone informed of progress. These activities were essential to helping us pull through norming to become a high performance team.
We also used this data as the basis for a more holistic study as shown in this team reflection super graphic. There's some prose and a video walking through the graphic along with additional reflection on our team blog.

Lots of teams try to measure morale or unity. Unlike most methods, Lencioni’s survey is based on sound research and gives concrete feedback on what you need to do to actually improve how your team works together. If you aren’t naturally having a blast coming in to work with your team, chances are there is something dysfunctional going on. I highly recommend taking a look at Overcoming the Five Dysfunctions of a Team book.

Boxing the Infinite Design Space

2013-04-15T09:00:00.000-04:00

When I work on a puzzle I start by making piles of pieces that seem like they go together. When I find a distinguishing feature on a piece (like an eye or a unique color for example) it helps me fix a position and I can start connecting pieces together. If I can figure out where a pile or set of connected pieces should be relative to one another I try to place them approximately in the right spot on the table relative to the other piles I've made.

The most important pieces are the corners. The only problem is that there are only 4 corner pieces (out of thousands) and in some puzzles it's impossible to determine just by looking at the piece, in which corner that piece belongs. Edges are helpful too but without some kind of context it's difficult to know if an edge belongs on the right or the left, the top or the bottom.

With each puzzle piece slowly being discovered and placed, I am slowly exploring the available options for putting the puzzle together. By identifying essential landmarks, I'm defining boundaries that let me explore within a specific context without venturing too far into the weeds.

Jigsaw puzzles are technically finite (there's only 1000 pieces in a box), but I wonder if there is still something to learn from puzzle building strategy when approaching a software design problem with theoretically infinite possibilities. How can you quickly explore the design space so that you focus your effort on the design decisions that matter most? Is it possible to efficiently draw a boundary around the seemingly limitless design options of a software system?

Boxing in the Design Space

This is the start of a workshop idea aimed at "finding the corner pieces" with the intent of developing a more systematic strategy for exploring the plethora of solutions that can exist within the design space. I haven't tried this workshop yet, otherwise I would provide some specifics on facilitation and examples -- so if you try it out please let me know how it goes!

Exercise: Quickly and collaboratively create 4-5 designs that are built to the extreme for specific, desired system properties and still achieve required functionality. The general idea is that you would work in pairs or small groups to "turn the dial to 11" on specific quality attributes.

Outcome: With these designs in hand (at least one per quality attribute) you now know what the outer edge of the design space will look like for any given property. You cannot go beyond the extreme. Your design lies somewhere in the middle as either a blended solution or some combination of boundary designs.

This workshop has some possible key advantages.

The outcome of this workshop should help define the furthest extremes, effectively creating a boundary around what is realistically possible.
In some cases you'll be able to expose tension between designs and this tension will be more obvious because you have concrete examples demonstrating how some system properties conflict.
At the same time you will likely have concrete examples of mutual promotion -- cases where the same design can promote different quality attributes.
In addition to boundaries, the various designs created also enumerate some alternative branches that might be possible. For example, by identifying tactics that satisfy a quality attribute.

There are also some possible disadvantages to this workshop, reasons why it might not work as hoped. The biggest one is that in all likelihood, the best design is not a direct combination of "best fits" for system properties, but instead might be an altogether different design completely. Gordon Glegg shares some poignant examples of this common phenomenon in his book "Selection of Design". In his book, Glegg points to the evolution of the gas engine as an example in which revolutionary design was required to find a satisfactory solution - evolution or the naïve combination of "best designs" actually resulted in worse engine performance.

So while this workshop may help to bound the design space, and in many problem domains for software it will likely work extremely well, when revolutionary design is required other methods of exploration may be needed.

Play Telephone with your Architecture Description

2013-03-13T09:00:00.000-04:00

The game starts with a simple phrase.

I heard that Sally ate nine cooked crabs and a plate of shoestring potatoes but later decided that eating crabs was wrong.

You tell the next person, quietly so no one else can hear.

I heard that Sally ate nine cooked crabs with a plate of shoestringed potatoes and decided eating crabs is wrong.

And that person tells someone else.

Sally ate nine cooked crabs with shoestring potatoes and she thinks eating crabs is wrong.

Who tells someone else.

Sally ate nine cooked crabs with sho-shu potatoes and she tin cans wrong.

And in theory, the last person hears the same message as the first person.

Ally ate nine crooked crabs with shoo-shoo potatoes and something about tin cans singing songs.

But it almost never works out that way in practice.

That's the "Whisper Game" or "Telephone" as we called it growing up.

You play the same game on software projects, especially when using lightweight design descriptions such as the system metaphor or architecture haiku. These lightweight design descriptions are intended to be shared by way of an architecture oral history. They are supposed to be living representations of the design for the software system being built, passed directly from one developer to another.

While playing Telephone with your system's architecture doesn't initially sound like a great idea, it's actually a really good thing. Playing Telephone lets you know some important things about your team and the architecture of the system being built.

Is your team aligned? Ask any two teammates about a specific design decision. If you get two different answers then you know your team is not necessarily on the same page. A paper artifact can give the illusion of alignment while in reality the team may not all understand the architecture of the to be built system.
Have you explored all options? Letting the design evolve as decisions are passed by word of mouth, from developer to developer, creates opportunities for exploring options. Each time a developer tells the story of how the design satisfies specific properties desired in the system is a chance for uncovering new and better design decisions. This is an effective way to challenge assumptions and overcome bias when only a few individuals are directly involved in the design.
Does the design stand up to scrutiny? Sometimes individuals hold back when the only option is to raise concerns in front of a large group. Bad design decisions that might go unchallenged in a larger group setting won't stand up to scrutiny among small groups of peers. Pairs and threesomes can easily play devil's advocate without feeling the fool. The result is a strong architecture that holds muster to real challenges, not just straw-man arguments.
Is everyone learning from the design? A document can only describe a design in one way. Allowing the architecture design to be described through story telling lets the story teller adapt the story to best meet the needs of their audience. As a result, teammates learning about the design stand a better chance of learning and internalizing the key decisions they will be responsible for implementing.
Does everyone own the design? A formal architecture description written on paper is really only owned by one or two people. Whether this ownership is official or not, the authors of the document will know the design the best and have the greatest vested interest in ensuring that the architecture is satisfied according to their design. Allowing the architecture design to be told and retold allows everyone to become an author. Now everyone has a vested interest in achieving the design - everyone is an architect.

Effectively communicating design decisions to a group of people is challenging. Writing things down certainly has its advantages. In reality, few individuals will take the time to read and really study an official architecture description anyway, so having a solid architecture oral history strategy is necessary for ensuring your team communicates and develops software as effectively as possible.

Yes, the design will evolve as it is communicated informally from person to person. But what is worse -- playing a game of Telephone with your architecture or playing with your architecture all by yourself?

High Ceremony Software Architecture Kills Collaboration

2013-02-26T08:00:00.000-05:00

I was a panelist on the "Architecture and Collaboration" panel at the SATURN 2012 conference in St. Petersburg, Florida this past May. For the panel I was asked to provide a single slide stating my position. This was my first time sitting on a panel and the first time I was ever required to have a clear "position" on a subject. It took a long time and a lot of thinking before I finally was able to A) understand what my own position was, B) have a position that would differentiate me from other panelists (we all by and large have the same opinions toward architecture and collaboration), and C) articulate it in such a way that I could fit it on one slide.

My position was simple, clocking in at 4 words: "High ceremony suffocates collaboration."

Assuming the "voice of the outsider" for the panel and representing an Agile perspective, I explained my position (out of nerves, somewhat poorly) at the start of the event. Staring at 150+ faces in the crowd, I could feel the shock, horror, and confusion from many in the audience at what I was proposing.

My fellow panelists and the audience immediately pounced. "What do you mean 'high ceremony suffocates'," they cried, "because the methods we use help us collaborate!"

We are surrounded by ceremony when it comes to software architecture. The vocabulary. The abstract concepts. The workshops and processes. The artifacts and documents. Some subset of these things is supposedly required for effective software architecture design. Formal checkpoints and documents are intended to improve a team's chances of building the right thing on time and on budget by promoting collaboration among teams and individuals. While some ceremony, like a common vocabulary and basic understanding of core concepts, is essential, sometimes the ceremony becomes the goal. As a result the things intended to help teams instead become blockers to effective communication.

The high ceremony I'm referring to are documentation, formal diagramming, multi-day workshops, "holier than though" arguments about vocabulary. These things can get in the way of effective communication and prevent true collaboration.

Collaboration is the act of working with others to achieve a shared objective. In software architecture, collaboration is all about creating a shared understanding. Documents do not create understanding. Reviews and checkpoints are an act in absolution ("blessing a design"). Workshops are a means to an end, not the goal itself. It is tempting to assume that because a design is formally articulated or because everyone participated in a workshop that this is enough, that the team is all on the same page.

From a collaboration perspective, the right methods are the ones that facilitate to most interactions per second among teammates. This might be a document. It might also just be a whiteboard. Workshops are certainly helpful, though the right workshop for your team and situation may not necessarily be the one with the most rigorous engineering process.

As architects we are the masters of ceremony. And this is a role of immense responsibility.

If collaboration is your goal, do not choose high ceremony methods. High ceremony is not about collaboration but rather pontification.

Overall the panel went extremely well. I successfully stirred up some controversy and started an excellent converstation. For that matter, the intelligent, lively discourse throughout the entire conference is a true reflection of the kinds of high quality software architects that attend SATURN. The SATURN 2013 conference will be held in Minneapolis, MN April 29 to May 3 and the conference program looks to reflect the same high quality as last year. Mary Poppendiek, Scott Berkun, Stephan Murer and Pilippe Krutchen are all giving keynotes

Reflection on Past Essays and the Path Forward

2013-02-16T12:16:00.000-05:00

After a few years of paid hosting, I’m moving my blog to Google Blogger. When I first bought hosting I imagined all of the interesting, amazing web applications I could build. Sure, I built a few things here and there, but over time the web evolved and the free tools available have improved greatly.

As a part of the migration I moved all posts and comments, doing my best to keep everything as true to the original post as possible. In the case of comments, I migrated the exact text you left along with your name and web address. Unfortunately this means your gravatar ID verification will be lost and nothing will be connected to Google or Google+ IDs. This really makes it obvious how difficult the whole idea of “identity” can be on the web. If you see a problem, let me know and I'll fix it.

You might also some notice some hiccups in the RSS feed (including a Twitter blast), and for that I apologize. Though, maybe thanks to these hiccups you will take a few moments to read some past posts. I’ve had a good time re-reading old essays and seeing how much I’ve changed over the past few years. And how much I’ve stayed the same.

With this in mind, and with it being a new year, I thought this would be an excellent time to reflect on the trends and ideas I’ve written about over the past three years. All told that’s about 40 essays worth of content.

Agile + Architecture. Over the past two years I’ve really shifted focus almost exclusively to Agile software development, software architecture, and what it takes to do a good job combining the two. It seems like nearly everything I write beats around the bush of these two topics.
Too Abstract? In some ways I’ve become less concrete and moved further from the “core” engineering practices. I think some of this is a reflection of the world I live in currently contrasted with the world I lived in when I started this blog. Let’s put it this way, it’s much more difficult to make a business case for hard core engineering practices when many of the teams I’ve worked with over the past four years seem to struggle with establishing a solid foundation.
Affordance Driven Process Design. Applying affordances to process was sort of a weird idea at the time I first published about it in 2009. Now I’ve developed a wildly successful workshop with Ari Font that we presented at Agile2012 and Pitt Agile. Our workshop gives folks concrete advice on how to use process affordances. Plus, human behavior and cognitive science seem to be the hot new things in the Agile community these days.
Beef with Software Craftsmanship. I had some major beef with the Software Craftsmanship movement for a long time. I thought the whole idea seemed kind of dumb. And don’t get me wrong, parts of it still seem silly to me – the mentoring metaphors seem extremely presumptuous to me. Even Luke Skywalker couldn’t declare himself a Jedi without Yoda first deciding he had passed his trials. Luke was lucky that a 900 year old Yoda was still around to teach him valuable Jedi lessons. Sadly, most of us don’t have a Yoda – they’re extremely rare. The craftsmanship mentoring approach seems like having a bunch of self-proclaimed Jedi running around picking up padawan as they please… and as Obi-Wan taught us it’s not that easy to mentor…. But I digress. I had major beef with the Software Craftsmanship movement, but now I really get where they’re coming from. I’m a little older. A little more experienced. I’ve seen a little more of the world and become a little wiser. Craftsmanship is really about caring. Caring is essential for awesomeness; even the founders of the CMM recognized this.
Labels are Boring: A Common Theme. Speaking of craftsmanship, I’ve grown extremely tired of labels and how it seems to negatively affect folks’ thinking. When Demarco published his “Software Engineering is Dead” article, I was starting my final semester of a Master in Software Engineering degree. WTF, Demarco, why rock my world like that? Software engineering is an idea who’s time has come and past, but so what? It’s just a label for a way of thinking about the world of software development. But, while I’ve tried to move on from labels, there is still a lot of power in labeling something. Engineering. Craftsmanship. Agile. Architecture. None of them are really proper labels for the things they describe. As a result it seems I move between the labeled things, trying to blend them together in ways that folks typically don’t try. Knowing that I’m pulling ideas from a label is powerful – being bound to the label and letting it dictate my future is not.

The Future of this Blog

My original goal was to write, on average, one well-conceived essay each month. Writing 40 essays over three years hits the average, though significantly tapering off over the past year and half as I have is dissatisfying and not in the spirit of what I set out to accomplish. While it is a little disappointing, at the same time I’ve increased my participation in the local and national software development communities, generating original papers and workshops. This is certainly in the spirit of my intent here, though, these contributions have almost no public record here.

There are a couple of things I’m pretty excited about trying to finish and flesh out. Hopefully by making a public declaration of their existence, I will be likely to actually do it.

I always intended for there to be a “part three” to my series on “exploring the design space.” This third essay was planned to unify the first two parts by describing a risk-based model for pruning a multi dimensional decision tree. If that sounds a little odd, you can imagine why I haven’t circled back to complete it yet.
I am sitting on a ton of original research coming out of Ari Font’s and my workshop on process affordances. I think there are some interesting lessons and feedback waiting to be discovered here.
I’d like to do more original research, no matter how “amateurish” and “practitioner-focused” it may be. I miss writing about real software engineering practices. My first published experience report was on using science and experimentation to understand and evolve process. I should be taking my own advice on this one.
I should write more code, not just prose. Concepts are good. Seeing more concepts in practice is better.

So, was starting a blog a good idea? I think so. Seeing my thinking evolve over time has been fascinating. And it seems that a few people have learned from my essays – or at least I’ve sparked a few ideas and inspired folks to think from different viewpoints about engineering software. Critical thinking is challenging. Writing thoughts down, satisfying. Publishing them, kind of scary. Reading them years later, fun.

See you at the Affordance-Driven Process Improvement Workshop at Agile2012

2012-08-11T16:00:00.000-04:00

Here’s an introduction to our workshop at Agile2012. This is going to be an entertinaing and educational mix of theory and practice on Wednesday, August 15 at 9:00am in Texas 3. Learn what makes Agile tick and how to design the right process for your team! And please show your interest on the conference schedule or twitter!

Why do the processes and practices we use in Agile software development work the way they do? How do you know when to apply one practice or technique over another?

These are interesting and important questions for the software engineering community, but I'm going to cheat a little and give my answer to you in the form of another question: What does an emergency exit, a urinal in the Amsterdam airport, and a burndown chart all have in common?

At first blush not much, but let's look at these artifacts from the perspective of an ecological psychologist. When an ecological psychologist sees these artifacts, he'll think about how they fit in with the environment around them and how the artifacts help or hurt the animals (humans in this case) in that environment. Each of these artifacts serves very specific functions, but what differentiates one door from another or one urinal from another is how those artifacts promote specific qualities valued by the humans that use them.

Object designers might look at these artifacts from a different perspective still. Each of these things are specifically designed to achieve explicit reactions or cause a specific outcome from the people that interact with them. This is no longer about survival or opportunity, but about directly influencing decision making.

Built into each one of these objects are affordances. An affordance is a perceivable object or idea that directly influences a person's thinking toward a specific action. Let me show you what I mean.

Emergency exits are painted red and are opened by pushing a large horizontal beam on the door. Culturally, at least in the United States, red means "warning" and it is also very different from how other doors are painted. The horizontal beam looks like it was made for pushing the door outward to open it.

The urinals in the Amsterdam airport aren't all that different from the ordinary urinal. Most guys know what to do when they see one. Though we know what to do, sometimes we have a difficult time, let's say, executing flawlessly, resulting in some spillage. A true annoyance to janitorial staff and fellow bathroom patrons. To encourage concentration and truer aim, these urinals have a picture of a housefly etched into the bowl, near the drain. Urinal researchers (no joke, they exist) have shown that this fly etching encourages dudes to pee where they're supposed to anyway.

Burndown charts are simple graphs that show a team's progression toward fulfilling a goal. Usually these charts will show simple things like the hours or points remaining in an iteration to achieve commitments. What's amazing is that such a simple chart can convey so much information and gently alter a team's behavior. Something as simple as the slope of a line can nudge behavior towards greater productivity, sustainability, or even predictability.

Etchings of house flies, red doors, horizontal bars, the slope of a line -- these are all examples of affordances.

Here's the cool thing, as I hope you have already picked up. Affordances are built into not just objects, but process too. The best processes are the ones that encourage your team to naturally do the right things at the right times. Of course, amazing processes like this don't happen by accident; they are specifically designed to encourage desirable behavior while discouraging harmful behavior. By carefully choosing the process's affordances -- practices or artifacts that direct our thinking toward a specific goal -- a team can tailor a process that makes success intuitive.

This Wednesday morning (9:00am in Texas 3) at the Agile2012 conference, Ariadna Font and I will be exploring these ideas and putting the theory to practice during our hands-on workshop, Affordance-Driven Process Improvement: Designing a Process that Works for your Team.

The goal of this workshop is to not only introduce you to affordances for processes, but also to find true answers to the questions I asked at the beginning of this post: Why do the processes and practices we use in Agile software development work the way they do? How do you know when to apply one practice or technique over another?

See you there!

Scarce Resources: The Hidden Architectural Driver

2012-06-25T09:00:00.000-04:00

All things equal, the difference between success and failure on a project will be determined by how well you understand and deal with the scarce resources.

If a design, particularly a team design, is to have any conceptual integrity, one should name the scarce resource explicitly, track it publicly, control it firmly. – Fred Brooks, "Inches, Ounces, Bits, Dollars – The Budgeted Resource," The Design of Design

Scarce resources are hard to come by and in high demand. They are the ones that, if they were to suddenly become unavailable, could threaten your ability to deliver promised functionality and quality on time and on budget. Of course, this could describe a lot of things on a lot of different projects. Calendar time, engineering hours, system memory, domain knowledge, number of calls to a web service, CPU time, money, time in a day, bandwidth, database queries – depending on the project, any one of these things could become scarce enough that you need to pay extra attention to how you deal with it. Nearly anything could become scarce.

To make matters worse, scarce resources are moving targets and could change at any time. Different design decisions can make resources hard to come by or even free resources from scarcity, making things that were once scarce, plentiful. Architectural integrity demands constant vigilance from all team members.

Though scarcity is rarely mentioned explicitly, scarce resources drive design decisions in the architecture of a system, as much as functional requirements, quality attributes, and technical and business constraints. Generally there are only 1 or 2 scarce resources on a project at any given time and you'll nearly always want to strictly budget and track the most limiting, most scarce resources you know about.

Planning for Scarcity

To deal with scarce resources you must first identify them. The easiest way to uncover scarcity is with a little risk management. My personal preference is to make identifying scarcity a part of the team's regular continuous risk management routine. Combining the other architectural drivers with your team's threshold of success makes it easy to identify and prioritize risks that may influence the architecture. Looking for themes among the highest priority risks will uncover possible points of resource scarcity. Pay close attention to the technical and business constraints as well as quality attribute scenarios that seem especially difficult. These are areas where resources are likely to be scarce.

Most of the times your gut is a good indicator, but translating your gut feelings into something you can talk about as a team is tough.

Since resource scarcity evolves, it's not so important that you identify everything perfectly. It's much worse for you to do nothing than it is to measure and control the wrong things. Usually you'll need to identify some kind of a proxy that allows you to see and measure the scarce resource. And since scarcity tends to move around, it's ok (actually it's a great thing) if you refine your tracking and control methods as the project evolves. I've found that dealing with scarce resources is more like horseshoes and hand grenades than it is brain surgery – as long as you're focusing on the right general area it's better to do something than abstain from action out of worries about a lack of precision.

Looking for tension in quality attributes is another source good source for locating scarcity. Tension within the needs of a system is an indication that design decisions are experiencing influence from conflicting properties. Relieving this tension may not be possible, but understanding why it exists and managing scarcity around it is essential for satisfying all properties involved.

Resource scarcity should be included in teams' architecture discussions alongside functional requirements, quality attributes, technical constraints, and business constraints. How you choose to budget a scarce resource will influence how you design a system. Missing a scarce resource could create conflicts and prevent you from achieving specific qualities in your architecture.

Examples of Scarcity

Often times it's easy to get a feeling for where a scarce resource will probably be, but figuring out how to name, track, and control the resource explicitly can be much more challenging. In some projects, the scarce resource can be pretty obvious. In "Ten Pounds in a Five-Pound Sack" from The Mythical Man Month, Brooks talks about how storage (memory) was scarce in the design of the OS/360 and some of the lessons he learned when trying to budget that resource.

From my own experience creating Velocity search applications with Vivisimo, CPU is nearly always a scarce resource. When it comes to quickly crawling and indexing vast amounts of data (on the order terabytes of data, millions of individual documents), CPU cores are always in high demand and seemingly short supply. Identifying processing capability as a scarce resource gives us the upper hand in designing the system. We know to ask pointed questions about how often data changes, how much lag time is acceptable until changes are reflected in the search index, and how long it takes to crawl a complete data set. Carefully scheduling crawls is one strategy we use to manage this scarce resource. We also spend time profiling crawl efficiency, especially how much time is necessary to prepare documents for indexing. Shaving just 50 milliseconds off the time required to crawl and index a document can shave over 2 days off the total crawl time for a data set of 4,000,000 documents. Knowing that CPU time is scarce, we know that we need to spend more time thinking about the design of the system with respect to this scarcity.

Another great example can be found in The Xbox360 Uncloaked" a case study by Dean Takahashi. Scarce resources for the Xbox360 project included things like heat, the size of a silicon wafer (a proxy for a number of factors including performance, future earnings potential, and improved graphics quality), game developers' mindshare, calendar time, and physical size of the box. Juggling these scarce resources was essential to the project's success.

Practical Scarcity Management?

While I understand that scarcity is an important and often overlooked architectural driver, I don't have any definitive concrete guidance for identifying and managing it other than the nuggets of advice based on my own experiences and the few examples that I have shared. If these ideas resonate with you and you have more examples, please share them in the comments to help flesh this idea out further.

If you aren't Agile… Then what are you?

2012-05-18T09:00:00.000-04:00

If you tell me you are an Agile team, I assume that you are following, or at least trying to follow, the values and principles outlined in the Agile Manifesto. And I can easily verify how true you are being to those values and principles and to what extent you are achieving agility pretty quickly. Usually a 5 minute conversation about your team is enough to know where you lay on the agile spectrum. The Agile Manifesto creates a sort of measuring stick against which you can evaluate the agility of a team.

Even if your team isn't practically perfect, the values and principles from the Manifesto are your goal. I know that you are doing your best to embrace agility, but sometimes the real world gets in the way.

While participating in the "Architecture and Collaboration" panel at Saturn2012, Rick Kazman made an interesting comment that I let slide at the time. In response to the difficulty of getting stakeholders together to talk about architecture, Rick commented, in summary, "It turns out this is pretty easy. We all have the same goals. People want to make the system work."

This got me thinking about my development life, with and without Agile. Sure, everyone has the same goals – or do they?

The Agile teams on which I've worked had a clear goal, a clear purpose – to deliver value to our customer in the form of working software, quickly and predictably. Every decision that we made, no matter whether we were stuck in the weeds of a difficult technical problem or discussing high level methodology concepts, was made with this goal in mind. It always comes back to the software and how our actions would affect our ability to help our customer.

The teams on which I've worked that were not Agile, on the other hand, have always felt rudderless in retrospect. We had the same general goal of delivering software, but the focus on value was lost somewhere. The end result was usually disastrous. Little or no focus on quality. Process was either non-existent, which made the team unpredictable and unreliable, or so onerous that it actually prevented effective work. Some of these teams had fallen into the trap of jumping on the agile buzzword bandwagon without truly believing the values. Others were simply ignorant of the ideas.

On both the Agile and non-Agile teams, everyone did indeed have the same general goal of making the system work, as Rick pointed out. The teams that didn't actively embrace Agile ran into problems with how they went about achieving that goal. Without a common set of core values guiding the team, team members lacked the tools necessary to reach the goal effectively. It was like knowing we wanted to go north but not having a compass to show everyone the way. One person would use the stars, others the sun, and still others the moss on a tree.

The Agile Manifesto is a guide, a compass, a rough measuring stick, a rallying point for teams who want to build great software effectively. The Agile Manifesto for software development gives teams... attitude. And it's a flag for letting others know what your team stands for and how your team chooses to work.

I'm not saying your team is hopelessly lost if you don't subscribe to the ideas behind the Agile Manifesto. I've been successful with and without practicing it. The difference between the teams that were successful and the teams that weren't, was that the teams that were successful had a core set of common values uniting the team while we worked toward our goal. Embracing Agile's values and principles makes it easier to start this conversation and is a great starting point.

The big problem is that most teams who aren't Agile don't bother thinking about this kind of stuff.

Without a set of core values and principles, how the project ends up is anybody's guess. No guide, no compass, no measuring stick for understanding how well your process is working for your team. There is nothing for your team to rally around other than perhaps some corporate sayings or a strong personality on the team.

So this begs the question. If you are not Agile... what are you?