If I'm honest, from the day-to-day perspective, 2025 felt like a non-stop stream of uncertainty and struggle and just...ugh.

It came from all directions.

The general economy was...let's go with "choppy" - with tariffs and shutdowns and budgets cuts, oh my. AI rapidly morphed from a new and exciting efficiency booster to a possible existential disruption to our entire field. And in our own little world, Highline battled with a bit of an identity crisis - who are we and, more importantly, who should we be?

It was all a bit discombobulating.

That's not to say everything in years prior has been rainbows and unicorns. We've had trials and obstacles. But 2025 was a struggle. No doubt about it.

Isn't that what makes the journey meaningful though?

With the benefit of a few days of rest and reflection, as I look back at the year, I'm probably more proud of this year than any other in terms what we've learned and built, how we've adapted and changed course, and how we've stayed true to what we believe in and we're all about.

Here are my three favorite highlights from this year...

Highlight #1: In January, we launched Highline Coding Academy. After the pandemic forced so much of our lives into the virtual realm, our hope was to bring people of all ages out into the physical world to connect and build and learn and have a little fun. In this first year, we offered 21 different events and classes, facilitated by about a dozen different presenters and teachers, and welcomed 500 unique registratrants. Pretty great start.

And it was never about the money - which is a good thing, because (shocker!) we learned quickly that the economics of in-person education are not what you would call "strong". Instead, it was about planting the seeds of inspiration and creativity, and allowing for the serendipity of real world interactions. In terms of both of those, our little Coding Academy was a big success. Hearing from a few kids about coding projects they started on their own was a delight. And our professional events allowed us to make some connections and re-connections with friends and other like-minded builders.

Lastly, it also gave us a chance to eat some amazing Rockaway pizza on the regular.

Highlight #2: Bootstrapping a company isn't easy. 7 years ago, when it was just me, I was stressed to the MAX about what a late or no-payment would do to my family's personal finances, or what would happen if that next contract didn't come through. And this stress obviously didn't abate as we on-boarded employees and contractors over the ensuing years. Understanding and managing cash flow is a scary thing - especially when your degrees are in philosophy and information systems, not business.

In 2025 though, after 6 years of business frugality, Highline finally got to a place of solid financial stability, and all without any investment or debt. With the help of some great advisors (and a few business self-help books), I now lose ZERO sleep about late or no-payments - though, to be clear, I still want us to be paid!

This stability allowed us to do some new things - for example, this year we put into place a concept called "bench insurance", where full-time employees pay a small percentage of their billable rate into what we call a "bench pool", which we manage in a prescribed and transparent way. We've since accrued enough in our pool to allow for a guaranteed month worth of bench compensation for every employee, which gives a little peace of mind when contracts wind down.

Highlight #3: On the delivery side, while I can't share too many of the nitty gritty details, in 2025 we delivered some truly great projects for our clients. In the spring, we assembled a strike force team to build a mobile app and admin console for a client in the health care space. The timeline was extremely tight, the scope a bit murky at the start, and the deliverable crucial for their business. Highline delivered on time, on budget, and even took on a bit of extra scope. One of the best team efforts I've been apart of.

For other clients, we contributed to dozens of releases of different applications, services, and pipelines that helped their businesses in innumerable ways - like eliminating manual data entry for office staff, giving and tracking refunds for customers, helping to streamline the delivery of food to patients in long-term care facilities, and a ton of other great things.

TLDR; we built some really cool shit.

Well, now I'm on a roll...but I'll stop at these 3...though I have to say, I'm also super grateful for the awesome new people we've welcomed on board in 2025 as well as the equally awesome people we helped transition to full-time roles at clients.

Maybe it's just a function of this cozy time between holidays when work ebbs and stress fades that's adding a degree of sentimentality, but honestly as I look back, 2025 was, after all, pretty great. Despite all challenges, we persevered.

It wasn't about revenue numbers or profit margins - it's that we continued to build quality software and also stay true to what we're about: rewarding those doing the work and enjoying the collaborative process with people around us.

No matter where AI takes things, we'll be siding with the humans.

In the end, I'm super grateful for everyone who took a chance on our little company, for all the great work they do, and the professionalism and commitment with which they do it.

Happy New Year!

“The point of the organization is to enhance the lives of the people who are part of it.”
--Ari Weinzweig

Highline is a bit different from traditional consulting companies in terms of, well, everything - how we’re structured, how we compensate, and what we believe. That’s not to say that we’re “right” and everyone else is “wrong” - it’s all about trade-offs. At Highline, we value choice, autonomy, and freedom, but that can come at the expense of some other “good” things - like predictability, consistency, and structure. Plotting Highline versus other “gigs”, it might look something like this:

As you read on, it’ll become more clear why we put Highline where it is on the line above. And again, neither left, right, or center is “best” - it’s all about a person's preferences, circumstances, and goals.

The Premise

To start, it’ll be helpful to begin with the basic premise of our company, which is this:

Reward the person doing the work

For good or bad, most consulting companies run off the “billable hour” - revenue is generated by clients who pay for hours worked by consultants. And since it is the company who handles the terms, contracts, and payments, it’s the company that gets to decide how to divvy things up. Typically, the split looks something like this:

For every hour billed, consultants take about 50% of their hourly rates in terms of salary, bonuses, and benefits, and the money remaining is used by the company for a variety of things, like supporting back office functions (sales, marketing, recruiting, accounting, etc.), providing tools and office space, and paying out bonuses to investors and executives. Note that this split can vary depending on a consultant’s seniority, the domain or size of the company, and other factors. And again, there is nothing inherently “bad” about this model. Our goal with Highline though is to show that while the traditional model does have a proven track record, it doesn’t necessarily have to be this way.

As an alternative, Highline’s endeavors to have the picture look more like this:

First, we aim for our consultants to make a much larger share of their bill rate (closer to 80%, plus or minus), and also for them to make a larger absolute value than they would make at a more traditional consulting company. This is based on our belief that if you’re the one out there on the front lines working with clients, then it’s you who should take home the majority of what the client is paying for your hour. Secondly, as a competitive advantage, we try to keep our hourly rates less than other companies in our space (by about 20%, plus or minus). Putting these two things together - paying a bit more (and therefore attracting and retaining great people) and charging a bit less - we hope it’ll make for a compelling case for clients to consider and continue to work with us. And this has been the case so far.

Of course, you probably realized that there’s an obvious consequence to this - there's less money left over for the company itself to run the business. This is real. And so we cut out the frills, eliminate the bureaucracy, and run as lean as we can. In practice, this means that many traditional business functions are completely absent at Highline (like sales or marketing), and other back-office functions are handled by consultants themselves (as "extracurricular roles", which are compensated).

This model - pay more, charge less, and run lean - as basic as it is (i.e. “duh”), so far has yielded some great results. Over the past few years, we’ve assembled an amazing group of individuals (over 20 at the moment), built a fun and healthy culture, and delivered quality software to our clients which in turn helps their businesses thrive. So we’re off to a good start in proving that our Premise is viable. But there’s more to what we’re doing than just this.

Values

“The place to improve the world is first in one's own heart and head and hands, and then work outward from there.”
-– Robert Pirsig, Zen and the Art of Motorcycle Maintenance

Building off our core premise, we’ve begun to put shape to a few core values. These are things that not only make us successful in what we do, but just as importantly help us achieve balance and meaning in our lives.

1. Focus on the Work

First and foremost, we are people who believe in the work. We are builders, and creators, and tamers of complexity. Our focus is not on winning deals, growing accounts, or increasing margins. Our goal is to build quality software. That’s it. We believe the rest will sort itself out.

In practice, this value manifests itself in a couple of ways. First, as mentioned above, our goal is to reward the person doing the work, and so we aim to give consultants a larger share of their bill rate than they would likely get at other companies. Going further, at a cultural level, we consider the “heroes” of our organization to be the consultants grinding it out in the trenches for our clients, and not the sales people signing contracts, the strategists crafting new elevator pitches, or the execs twiddling with spreadsheets. Our focus is building great software, whether that means writing robust code, designing delightful experiences, or managing teams to successful outcomes. This is the soul of what we do.

Second, Focusing on the Work means being pragmatic and solution-oriented. We know the process of building software is inherently complex, where constraints are few, expectations are high, and solution spaces are limitless and malleable. Given this, we expect frustration and even thrive in it (on our best days, at least). Rather than getting tangled in the problems, tensions, and the ever-present risks, we calmly and resolutely focus on the work, and find our way to a workable solution.

2. Keep it Simple, Streamlined, and Flat

As mentioned in the Premise, giving consultants the lion’s share of their billable hour means that there’s less leftover to run the business. For Highline to be successful then we must keep processes, organizational structures, and policies as simple as possible. We literally cannot afford bloat. (Also, we hate it).

In practical terms, this means that there are no back-office positions, and basically no hierarchy at Highline. Everyone is a Consultant. Some consultants have more experience than others, and can therefore command higher bill rates (and compensation) or more senior roles on a given project. And some consultants may even take on “extracurricular roles” which give them responsibilities (and compensation) for helping with operations. At the end of the day though, within Highline, we are all peers.

It’s important to note that to function effectively without hierarchy, we must rely on both less structured lines and forms of communication and collaboration, and also on the assertiveness of individuals to navigate our network to get what they need. This means that we drop 1-1 meetings on each others’ calendars, initiate impromptu working sessions at coffee shops (or breweries!), and send important and unimportant (memes!) information over Slack to keep eachother in the loop. Rather than a command-and-control organization, we are more like a decentralized bee colony, each individual focused on their task, heeding common rules and norms, minding others, and getting work done.

3. Listen and Be Humble

We are not the consultants who label themselves as “thought leaders”, who push the latest trendy processes or technologies, or who spend meetings talking about ourselves and our methods. Quite the opposite. We are listeners and learners. We have confidence in our experience and abilities, yes, but that confidence is accompanied by a humility in knowing that the best solutions are a composite of ideas, suggestions, and compromises from all stakeholders. In other words, there is wisdom in the crowd, and we stay open and attuned to it.

On projects, we start by asking questions. Lots of them. Our aim is to understand the context and contours of the problem space as much as possible before we break ground. What we know is that it’s not just about building software right, but about building the right software, and to do that requires listening. As we do inject our expertise and vision into our projects, we do so without being wedded to our own solutions. Software development is about trade-offs, and our goal is to present options with pros and cons, and help navigate and execute toward the best path.

4. Choose Quality, Lifestyle, and Balance

Finally, we are human beings. We love what we do, but we also have lives outside of work - family, friends, hobbies, and commitments. And so we strive to keep it all in balance for the sake of our happiness (and sanity). As a natural and fortuitous consequence, the balance and calm we carve out for ourselves also benefits us in our work. Rather than being frazzled and time-constrained, we have space to be thoughtful, creative, and productive for our clients.

Further, in support of balance, we try not to overextend. This goes for us as individuals and as well as for Highline as a company. As consultants, we take on only as much as we choose - whether that means 20, 30, or 40 hours a week - and step away for vacation and time off as much as we’d like. (This comes with a trade-off for earning potential, but it is our choice.) As an organization, we don’t take on projects for the sake of “growing the business”. If we believe we have the skills and capacity to succeed on a project, we will. If not, we’ll pass. We don’t chase growth for its own sake or for the sake of profit, but we do consider and accept it if it makes our collective group more sustainable or gives us more choice.

Conclusion

I am really proud of what we've built at Highline over the past 6 years. It's not perfect, but we try to take a growth mindset on our journey - accepting that we're not going to get it all right from the start, but learning from our mistakes and incrementally making things better.

Happy to hear your thoughts, or if you'd like to talk about software we could build for you, please reach out! We'd love to help if we can.

On the left is when we stay in our lane. We're given a requirement, we interpret the requirement, and then we make it so. Easy peasy. But what's that? The requirement didn't mention validation and it's a form submission screen? Must be it's not needed! We just do what we're told. And likely we're not this pedantic, but the point is that it's comfortable at this end. We get work done, and we don't make waves. If something isn't right, we were just following orders.

On the other end is when we go rogue. We find some "requirement" that might be useful (but no one really asked for), or some new approach or technology that's interesting or "hot", and then spend the entire sprint in our coding cave joyfully building it out. Will it actually help the project? Err...maybe? Regardless though, it'll be fun, because we love nothing more than smooth sailing - not being beholden to the ideas, thoughts, or reservations of others (well, that is, until they see our 4k-line pull request!).

Unfortunately, in neither of these states does valuable work get done. I mean...sure, something is produced, but it's rarely the right thing - i.e. the thing the users or the business actually needs. Because this "right" thing is really hard to figure out. It requires understanding the business context, priorities, and time constraints. It requires understanding the needs, preferences, and behaviors of the users. And it requires understanding the existing implementation and the scope and impact of what is to be built. All of this must be hashed through to get it "right", and this "hashing through" process is what happens in the Ambiguous Zone. It's messy and frustrating, but it is the only way to produce something of real value.

The most effective developers I've worked with understand this, and are adept navigating this zone. They are curious about the perspectives and needs of other stakeholders, and ask good questions. They push back when things don't make sense, but do so tactfully. And they understand the realities of building software in an industry context, where there are budgets, deadlines, and other less-than fun realities. They don't expect everything is perfectly spelled out for them (whether by designer, business analyst, or architect), because they know, as the one closest to the implementation, they have a crucial perspective to voice. But they also know they don't have all the answers. In other words, they know that building the right software is a back-and-forth, collaborative process.

But not every developer or team works well in this Ambiguous Zone. And there is a tell-tale sign when it's being avoided: the conspicuous lack of discussion. When we're either doing only what we're told or only what we want, we have no need to talk with others. It's all laid out for us, either in a requirements spec or in our heads. If we do happen to meet, there is no disagreement, no requests for clarification, no passionate exchanges. And while it's alluring to think this quietude is a sign of efficiency or productivity (meetings suck, amiright!), it's NOT. Instead, it's a sign the team is avoiding the Ambiguous Zone, and the end product will be off target.

It's understandable though why we would avoid it - it's both risky and frustrating in there! First, we know that there's a real chance we may never make it out. We'll flounder in meeting after meeting with no resolution, and never get to build anything. (At least when we do what we're told or what we want, something gets produced.) It's also very hard to operate within the Ambiguous Zone, because it means navigating the imprecisions and imperfections of other human beings. This was tricky in "normal times", but probably more so in a remote-first environment (without the benefit of hallway conversations or other in-person interactions that build trust and enable quick information flow). Without these discussions, we don't understand the context, we don't voice our perspective, and we often don't find the right solutions.

My sense is that the Ambiguous Zone is an inconvenient truth about programming in industry. We're often driven to the field because it's deterministic and precise, but when we get here, it's much more fuzzy than we expect. In other words, there's a mismatch between the perception and the reality. Just look how we interview. We ask candidates to solve esoteric programming puzzles (i.e. do what you're told) or assess whether they built some interesting side project (i.e. do what you want), but rarely do we evaluate whether they can navigate ambiguity in a collaborative setting, which is just important to being an effective developer in the "real world". As an industry, we should emphasize, encourage, and train for this more. Happy to hear your thoughts!

There are two projects, both deemed important by the business, and both need a UI developer. Unfortunately, only one UI developer is available. The manager has an idea though: why not just let the UI developer split his time across both projects. Sure, this will slow the pace a bit for each project, but at least they'll be moving in the right direction.

So what's wrong with this? While it's true that the UI developer should be able to contribute in some productive way to each project, what's faulty here is the hidden assumption. The manager typically expects in this scenario that each project will effectively be getting one half of a developer worth of productivity. In reality, both projects will get less than one half (and possibly much less). Here's why...

The amount produced by a developer can be calculated like this:

Productive Time = Total Time - Overhead

Total Time is the number of hours you're on the clock - say, 40 - and Overhead can be split into two pieces. The first is the Fixed Overhead - things we need to do as part of our job, like checkpoints with the boss, company meetings, etc. (see bullshit). This type of overhead just comes with the particular company we work for, and there's not much we can do about it. Assuming a Fixed Overhead of 10%, that would leave 90%, or roughly 36 hours, left over.

The other type of overhead that needs to be subtracted out is specific to the project, and so we can call this...wait for it...Project Overhead. This is the time we need to spend in order to just keep pace with the project as it's moving along, and includes things like reading emails/IMs, attending meetings (stand-ups, backlog groomings, etc.), and other basic communication and coordination necessary to understand the current state of things. And this is where the fallacy comes into play. We often forget that each project has its own overhead - work that needs to be done before we can start doing anything truly productive - and when you add all of these overhead activities up, it's not uncommon that this could sum to 15% or more of your time per project. Given the example above, this would leave the UI developer with only 60% total productivity, or 30% for each project.

And this is why each project team will get less than half of a full developer's worth of work.

Now it's important to note that Project Overhead is variable, and is primarily a function of how many people are on each project team. The larger the team, the more knowledge that needs to flow back and forth, and so the more time that's necessary to absorb this knowledge. On big teams, it could easily take 20% of your time to coordinate with your team before you can contribute something of value. And again, if you're splitting your time across two big projects, this premium would be paid twice.

The flip side, however, is that on solo projects, there is effectively no Project Overhead, and so the fallacy doesn't really apply. You can happily float back and forth between the two, because there's basically no switching cost (aside from the nominal costs of booting up your local dev environment, loading the project details into your brain, etc.). In my experience though, these types of solo missions are rare.

Finally, the problem is especially bad when a developer gets split across 3 or even more projects (which I have seen). This is pure insanity. After incurring the Project Overhead costs for each of the projects, the developer has almost no Productive Time, and basically can't get anything done. And it's not their fault.

So why do we fall for this fallacy? I think there are two reasons. First, I think it's fair to say that as a group, we developers like to get our hands dirty on lots of things. It enables us to learn more, which is something we enjoy, but also something we know is good for our career (and earning potential). And so we accept being split across multiple projects out of self interest and optimism. And full disclosure, I do this shit to myself all the time.

The second reason is that the ones making this decision for us to split our time are typically the managers, and managers aren't always tapped into the same reality that we live in. The reason, I think, is that a manager's work is Overhead (in the way that I've defined it). They keep up to speed with what people are doing and how projects are tracking, and this is incredibly valuable to the organization, but the need for them to have dedicated time to produce something is much lower than for other roles (which includes not just developers, but BAs, designers, etc.). And so because they don't require a solid chunk of Productive Time for themselves, they might forget that we do. Note: I'm just spitballing here!

In the end, I'm curious what people think. Do you agree that this is a thing? How pervasive is the strategy of splitting time across projects? Do we generally understand the effects on productivity? And are there other things at play that reduce productivity when splitting across projects?

Sure, there were the miscommunications around scope or requirements. Or differences of opinion on technical solutions or philosophy. Or even just general stress around tight deadlines. But in any of these scenarios, I always felt like "I got this". I could navigate through just about anything, at least without ruffling too many feathers or dropping the ball too egregiously. Things were comfortable, more or less.

Then I went out on my own. And immediately it felt different.

Of course I expected this to some extent, but I was surprised at the degree to which I felt exposed. It was as though I had found myself in some dark alley, danger lurking around the corner and no one to rely on but myself. I was Chuck Norris in some 80s action movie. Except without the martial arts skills or unflappable confidence.

This exposure came at me in two ways.

First, there were new responsibilities on my shoulders - things that were taken care of for me in my prior life. I now had to market myself, negotiate my own rates and contracts, accept and also turn down work, and so on. And this part was a feature not a bug. I expected that I'd need to do these new things, and in fact it was part of the whole appeal of me going out on my own. I'd need to be self-reliant, but in return I could control my terms - what I worked on, when, and for how much - and to not cede control of these things to some other person or entity who was also trying to optimize their best interests. It was new and stressful to do these things on my own, but I welcomed it.

But there was another dimension that I didn't anticipate, and that was the day-to-day exposure. Right from the start, the things that I was used to doing as a normal part of my job felt a bit different. I was accustomed to being a consultant, and so my whole approach was already infused with an extra ounce of tact and agreeableness. But it was different being solo. Before, I had always had at least a few people on "my side". Other consultants coding along with me. Client partners advocating for me. Project managers backing me up. Without that team, each and every interaction felt a bit weightier. Every assignment or task a bit more crucial. I was more conscious of how I held myself when facilitating a meeting, more aware of every code commit I made, and more vigilant about being productive always.

Working for a consulting company, there was always some inertia working in my direction. My small screw-ups or lapses, it always seemed, would just be rolled over by the greater machine, lost or forgotten. There were people to back me up, or share responsibility. As an independent though, what I did and what I produced was more directly linked to me and solely me. There was no cover. I stood alone. And this did ratchet up the stress.

So if the exposure, both in big and small ways, increases after going it alone, why do it? It seems like a net loss. Here's where there's a silver lining.

As many wise people have said in one way or another, without discomfort there is no growth. And for me, this has been true. These new experiences, though stressful, have been a great source of learning. Before, as a doer, I was never "in the room where it happened", and so never learned how to confidently negotiate, market myself, or navigate tricky account situations. Now I have some level of comfort with these things. I've added some business skills to my primarily technical toolbox.

There's another layer to this, beyond just the new experience. As a consultant for someone else, I always gave up a cut of what the client was paying for me in compensation for shielding me from these "exposed" situations. I'd kick up anywhere from 30-60% to pay for the client partner to negotiate for me, and the accountant to do my invoicing, and the project manager to generally grease the skids.

Often this seemed completely fair - these things were a big help, so I could focus on my work. But eventually, as I got more experience, this cut that I surrendered to someone else to do these "businessy" things seemed incommensurately large relative to how much effort they took. If someone helped me land a new contract, it made perfect sense to kick up a large chunk of my bill rate. But then when I got extended for a 2nd year, why was I still giving away 40% of my bill rate to that person who only helped in the beginning? Basically, there was a tangible cost to being shielded from these "exposed" business situations, and now that I'm on my own, I don't need to pay that cost, and it's nice.

So in the end, there absolutely was an initial bump in stress and discomfort with the new level of exposure. And, admittedly, I did drop a few balls and maybe even burned a bridge or two as I got my feet underneath me. But I've appreciated the confidence I've gained through these experiences, knowing that I can now handle myself in a few of those dark alleys.

This is part 2 of a series. Read part 1, The Loss of Bullshit.

Then I went independent, and all the bullshit just vanished. Poof. Now, my work hours are totally optimized. Everything I do, whether it's coding, or collaborating, or researching, is all "productive", in the sense that it contributes in some practical way to the project I'm working on. And so without a doubt, I get way more done now.

In addition, I find that I also enjoy the work more, because I thrash less. As a full-time employee, bullshit would be just randomly scattered across my calendar, and so every day would be a non-stop context-switch between bullshit and productive work. I'd come out of some useless meeting and then frantically try to squeeze in an hour of real work before I had to get to my next bullshit activity. Now, I have longer blocks of focus time where I can sink into the "zone" and do deep work, and it is nice.

So for someone who generally enjoys their work, all this is pretty freaking great.

But I've come to realize there is also a flip-side. Although bullshit is, well, bullshit from a productivity standpoint, it can sometimes be kind of nice from a personal perspective. Like a pressure release valve. After a few straight hours of coding, for instance, when I had effectively depleted my mental reservoir, sitting in on the quarterly meeting and listening to some exec talk about corporate financials could actually be a relaxing switch-up. A chance for my brain to take a break, and focus on something different. Or to just generally space out.

Bullshit could also be a forcing function to do something that benefited me in the long term - like networking or learning some other part of the business. Although in the moment it seemed like a distraction to my immediate goal, in hindsight, I can see that there was a little bit of value there for me.

And so some days, I have to admit, I lament my loss of bullshit.

Now of course at this point you might want to shout back: "you're an independent for god's sake, you can do whatever bullshit you want, whenever you goddamn want to!" And technically that's true! The big difference, though, is that bullshit is now on my dime. Because, by definition, bullshit is not tangibly productive, it's not something I can bill a client for. And so there's now an opportunity-cost to bullshit. Every hour of it that I do is an hour I'm not making money.

And so given this financial reality, combined with my original antipathy toward bullshit, it made it so, at least in the beginning, I would hardly ever opt for it. Eventually though a bit of productivity fatigue set in, and I realized some level of bullshit was actually healthy. So I came up with a few guiding principles to balance things out.

First, since I no longer get paid for it, I realize that I should at least get to choose what type of bullshit I want to do. So if what I actually need is a mental break, I just go mountain biking, which is way more rejuvenating than a company meeting, in my humble opinion. (And of course I realize mountain biking is a hobby and not bullshit per se, but that's kind of the point.)

Second, I try to frame it differently, and less rigidly. Some of those things I considered bullshit before, I now see as a type of investment in myself or my business (maybe even a bit like Extra). So for example, while an hour of browsing tech blogs or a random zoom connect with a former colleague might not immediately help my project, it can be a good way or to learn, to network, or just to be a human being, and so I deliberately try to make a little time for it, and not stress that I billed 7 hours instead of 8 that day.

In the end, I found that even bullshit has tradeoffs. While I do enjoy my current lack of it, that doesn't mean I don't sometimes miss an informal one-on-one with my boss or a random celebration. I will say though, I have not once missed those HR training courses.

--
* I realize that the things I'm calling "bullshit" are often valuable activities from the organization's perspective. Regulations that stipulate employees take certain trainings are real. Meetings that inform all employees of the state of the business are important. And so on. But from the perspective of the person who just wants to build quality software (or whatever their immediate job is), these things can end up feeling like a distraction. Like bullshit.

As a simple example, imagine a small team of, say, 6 developers:

The team has created and now supports a mobile app, an Alexa skill, three separate web apps, fifteen microservices, and a partridge in a pear tree.

And while they've managed to enforce some level of standardization across the ecosystem, by the very nature of modern software development, it's just a lot. There's Python, Django, DynamoDB, S3, React, Typescript, Tailwind, Swift, and much more, all working together to implement user interfaces, APIs, data persistence, integrations, and so on.

Too much for any one developer.

Moreover, each sprint comes with different demands for enhancements and fixes, and the work is rarely distributed equally across the codebases. One sprint might require heavy changes in the mobile app, and the next might need mostly back-end work.

So again, the question is: how is a team best deployed to fulfill the demands of the business, sprint-after-sprint? In other words, how do we best divvy up responsibilities?

For example, do we encourage specialization - say, assign Emily to handle all the mobile development, let Joe own the web components, and so on? And beyond just tech stack, if we have over a dozen microservices, do we draw some boundary around them, and say Javier owns microservices A and B, Anna C and D, and so on? But what happens when work in their specific area is light for a few sprints in a row? Are we comfortable with them having a little idle time?

Or do we try to better optimize utilization, and encourage a more full-stack, cross-domain culture? Each developer does it all! But wait...is this reasonable?

Drivers

Just like when building software, before we jump to a solution we should step back try to figure out what we're trying to solve. I'd argue, there are four big things to consider when thinking about developer responsibilities and ownership:

1. The obvious thing is productivity. All things being equal, we would like to structure the team in a way that maximizes the work they get done. I mean, this is why we're paid! And so optimizing productivity typically means keeping some consistency for developers, aligning their work with their skill-sets and experience. For example, if Emily worked on the mobile app last sprint, she should be able to knock out that next feature faster than, say, the developer who has been working in the backend.
2. We also want to be mindful of quality. And similar to above, it's likely that the developer with more hands-on experience in a given codebase will also implement the next feature with higher quality than the one who was parachuted in for one sprint only and doesn't know the structure, patterns, or conventions. Even the best developers can write crappy code when they don't understand the codebase.
3. Here's where it gets tricky though. The organization often also needs to mitigate the risk of a developer leaving, and with it their knowledge. And this often runs counter to the goals above. Whereas assigning one single mobile developer to all the mobile development sprint after sprint might maximize both productivity and quality, it puts the organization in a precarious position when that developer starts getting texts from the Meta recruiter.
4. Finally, we can't ignore the less-tangible dimension of developer happiness. But this too is complicated. Some of us like variety and newness, and so if we're stuck with the same system sprint after sprint, it could affect our productivity or make us more likely to jump ship. Others prefer predictability and take pride in having their own turf. Bouncing around would drive them crazy. In other words, each of us is different, but ensuring that we're fulfilled and challenged is important.

Now assuming these are the right factors to consider, the crux then is that every organization has to choose its "weights" for each. What's most important to optimize? Maybe there's some time-critical force (deadline) at play, and productivity needs to be optimized above all else. Or maybe the market for developers is so hot that the most important thing is just keeping developer happy, productivity be damned! In other words, it's very much context-dependent.

Hand waving over the "how" here, and assuming the organization knows what it's trying to optimize, it needs to pick some model for structuring responsibilities for the team. But what are the options? Here are a few of the popular models I've come across:

Ownership

Often, we settle on a strategy of code ownership. Sometimes it's tacit - everyone just respects that Joe usually does the web stuff, Emily does mobile development, I do the payment microservices, and so on. Other times it's formalized - management hired Joe as The Web Developer, Emily as The Mobile Developer, and so on. In either case, in practice it plays out similarly. When new features or fixes come in, we divvy it all up according to what we "own". This allows us to be maximally productive (in an individual sense), and also increases quality (usually). There are a few downsides however.

First knowledge of different systems can be hoarded, representing a huge risk if someone "wins the lottery"/"gets hit by a bus", as they say. Additionally, "ownership" can sometimes feel like a shackle to developers, if what they want to be doing is learning new things. Lastly, it must be noted that while individual productivity increases (since the developer knows their codebases well), collective productivity often decreases with ownership. As discussed above, because work loads don't cleanly balance out each sprint, this can lead to a feast and famine situation for the owning developers. For example, one sprint there's more work in their codebase than the owner can handle causing big bottlenecks (or late nights!), and the next month there's not much to do, and that owner is more or less idle.

Free-for-all

When an organization feels some of these pains of ownership, they often abandon any strategy at all and just wing it: i.e free-for-all. This system is as simple as it sounds. A manager sees a pile of work over there, and says "hey you, code-person, get after it!" And that's that.

While this strategy does ensure that we balance our overall allocations (i.e. Emily the mobile developer isn't idle when there's no mobile work, because she gets pulled over to work on the Python service), this model can be both exhausting and fraught with quality issues. If we don't have any long-lasting connection to the codebase we're working in, our work is guided only by expedience and ignorance. We shoe-horn a janky short-term fix in because either we didn't know any better, or because we don't really care much - we'll be in a different codebase next sprint! As they say, "no one has ever washed a rental car".

The free-for-all model is often driven by management who idealistically sees us as interchangeable handymen (i.e. the mythical Full Stack Developer) - quickly able to tackle with grace and ease any technical challenge, no matter what tier, system, or business domain. And of course this is bull-shit. For all but the most trivial systems or all but the most exceptional developer, there is too much complexity to be able to navigate our way through everything without proper time to ramp-up.

Stewardship

Now somewhere in between these two extremes of ownership and free-for-all is a model of stewardship (or custodianship). Similar to ownership, developers look after certain codebases where they have the most expertise - they probably did most of the initial work in the project, and their name is all over the code, but they're not responsible for doing everything. At the least though they must be consulted on changes and be pulled in to triage production issues or discuss design changes.

This allows for, say, Emily to go over and jump in and give Joe a hand with the web portal when the workload there is high, but to return to her mobile app when things die down. In other words, it balances both individual and collective productivity. Additionally, this model also can balance nicely risk mitigation and developer happiness, as developers build some level of familiarity with other pieces beyond their own (i.e. get to learn new things).

The important thing with stewardship is that, while it's less formal that ownership, it is still explicitly defined and held to. At a minimum, a few things should be in place. There should be a list that defines who is steward of what, GitHub (or whatever SCM tool) should be configured so that stewards must approve all pull requests, and stewards should be given enough allocation (time) to do the necessary "stewarding" of their component. Unless there's some formality and discipline around this, everything will just slide down the slippery slope until you find yourself in the pit of the free-for-all.

Conservatorship

Lastly, it's probably important to also recognize a model that's very popular in bigger organizations, and that's what I'll call conservatorship. This is the idea that there is a chosen few (usually astronaut architects) who have the responsibility of "owning" everything. They hold the knowledge, make the important technical decisions, and design the business logic/structure, but the actual down-and-dirty work is done by whichever developer is available, usually in free-for-all type of way.

In theory, this model works great. Efficiency and quality are optimized by virtue of conservators sharing their "wisdom" with the hands-on developers, thereby clearing the path to the quick and effective solution. And with the help of the conservators, developers can just swap in and out of codebases, relying on the conservator to get them up to speed quickly. That's the idea at least.

In practice, it never works out like this. Conservators, without any hands-on experience, quickly become detached from the on-the-ground technical reality, and often are no real help at all to developers who need to implement some enhancement or fix some defect. It can take a developer just as long to ramp up, and additionally can be frustrating because they basically cede their autonomy (and creativity) to the conservator who gets to call the shots.

Speaking as someone who has played this conservator role, I can attest to how little help I was to the hands-on developer who needed to get real work done within the systems I supposedly "owned". This model, in my opinion, sucks for everyone - and this is why I try to avoid these types of roles.

In Summary

In the end, what started in my head as quick post about different models of code ownership went way longer than expected. This is how I think about it though, and these are just a few models I've come across for divvying up work. I'd love to hear your thoughts and experiences.

* And yes, if you were wondering, I thought of the name "Conservatorship" after watching the Free Britney documentary with my wife. :)

There are three services: a Product service which manages data related to products offered, an Order service which tracks purchases of products, and a Catalog service which controls what products are published to different mediums (say, print, digital, etc.).

Now what happens if a given product needs to be deleted? Say "Widget X" is no longer a product to be sold. Or "Widget X" was accidentally added by some sleepy user, and just needs to be removed. What's the best way to handle this?

Solution 1: Just Delete It

In a monolith world, we might be able to just delete the "Widget X" row in the PRODUCT table and that would be that. If there was some foreign key reference to that product row from some row in another table (e.g. because the product was sold in a prior order, or the product was included in an existing catalog), there could either be a delete cascade that would automatically remove those rows too (often a dangerous thing), or at least there would be some foreign key constraint that would throw an error when trying to delete that product. Either way though, because all the data in a monolith resides in one database, the management of referential integrity (i.e. not leaving orphaned data) could be centrally managed.

In a microservices architecture, this obviously doesn't work. Sure, the "Widget X" row could just be thoughtlessly deleted from the PRODUCT table in the Product service, but other services would then have data that's effectively orphaned.

For example, the Order service might have a reference to "Widget X" in its ORDER_DETAILS table for an old order, and might want to fetch information on this product to show full order details. If "Widget X" was deleted from the Product service, the Order service would get a 404 on its GET request for /products/widget-x. In other words, even though each microservice is intended to be de-coupled and autonomous, it doesn't mean that it's completely independent from a data perspective. Coordination and communication is necessary if we want to preserve some level of referential integrity.

Solution 2: Synchronous Coordination

The next possibility is to have the Product service, on delete of a given product, first check with any dependent service to see if it's able to delete that product and then to carry out that "cascade" if necessary. For example, the Product Service could first check with the Order Service and Catalog Service if it's safe to delete "Widget X" (i.e. there are no references to it), and only on confirmation that it's safe would it do its delete from the PRODUCT table. Or alternatively, the Product Service could go forward with its own delete and then make a synchronous call to the Order and Catalog Services, and say "I just deleted my Widget X, so delete any reference to it from your databases".

Even if deleting this way was ok from a business rule perspective, this also is ill-advised. Creating a synchronous dependence from the Product Service to the Catalog and Order Services has major downsides - an increase in complexity, additional latency per request, and compromised resilience/availability (i.e. if the Order Service was down, the Product Service couldn't do a delete). Microservices should strive to be as independent as possible. Binding them together with runtime dependencies should only be done with great consideration, lest we just create a distributed monolith.

Solution 3: Just Don't Delete

At this point, we might observe that it doesn't really make sense to delete a product anyway! As Udi Dahan points out, in the real world there really isn't a notion of "deleting" things. Instead, the data just changes state. Employees aren't deleted, they're fired or they quit. Orders aren't deleted, they're cancelled. And Products aren't deleted, they're no longer sold. In other words, in almost all cases, it's better to support some status field on a given row than to outright delete it.

In the example here then, this would solve some of the problems. Instead of supporting a true "delete", the Product service would just expose a "decommission" (or whatever) endpoint for a product, but preserve the data in its database. In this way, the Catalog and Order Services would no longer have orphaned data - they could both still call back to the Product service to get information on "Widget X", even if it's no longer currently sold.

But this doesn't solve everything. What if the Catalog Service needs to be aware of a Product being decommissioned, so that it doesn't get shown to a customer for a given catalog? Sure, it could query the Product Service to get this information, but this could introduce a synchronous dependence from the Catalog Service to the Product Service, and introduce those same issues of complexity, latency, and resilience discussed above.

Solution 4: Asynchronous Update

To support the Catalog Service's autonomy, instead of relying on the Product Service it could maintain its own local cache of product data, keeping it sync with changes from the Product Service. When a product is decommissioned, the Product Service could emit a ProductDecommissioned event which the Catalog Service would listen for, and then it could update its own local product store. In the case of "Widget X", as soon as it was decommissioned, the Catalog service could know to not show it in its catalog. And this works. Except...

What exactly is this local cache of product data? Is it in-memory? Or a table in the Catalog Service's database? And how do we make sure it's in sync with the source of truth, the Product Service? There are a few options here, each with its own pros and cons.

If the local cache is in-memory, then this is light weight to implement, and potentially easy to synchronize (e.g. maybe there's some reconciliation process that checks the Product Service and ensures that its own data is fresh/up to date). The downside is that it gives up database-level referential integrity. If there's some product_id in a CATALOG table in its database, there would be nothing to enforce (at a database level) that that product_id is in fact valid (i.e. maps to a product_id in the Product Service database). Additionally, the in-memory cache may not tenable if the data to be cached is large.

Alternatively the local cache in the Catalog Service could also be a database table, which would solve the database-level referential integrity and storage issues from above. One problem though is complexity. Writing the SQL to seed and update a local PRODUCT table is not trivial - more code written means more code to maintain. Additionally, there's added complication around the synchronization - if there are multiple instances of a given microservice, but only one database instance, which microservice or what process is responsible for updating and synchronizing the cache?

Probably the better solution is to implement an Event Log (using Kafka, etc.), and let each microservice leverage this to keep in sync with changes to data. There is so much to consider with this option (way beyond the scope of this blog post), and a great place to start is Event Driven Microservices. While this pattern elegantly solves all of the problems above, this is, in my opinion, a level-up architecture, both to build and support, and so only something to jump into without adequate expertise and resources.

In conclusion, though I know this doesn't fully settle the issue on deleting data within Microservices, my hope is that it did highlight some of the salient issues and options. Please comment below with any tips or feedback!

One thing I've noticed over the years is that there is at least one thing that seems to be common to every good, veteran programmer I know. They all follow the same deliberate and dare-I-say selfish rule to how they approach their time: Always do Extra. Now before you call "bullshit", let me explain what I mean...

Assume there's a reasonable amount of work you need to do to fulfill the base expectations for your job (i.e. your Normal Work) and then there's a little time left over. (I know some may argue this "left-over time" thing, but just go with me). In this remainder time you have a choice: you can do More, Extra, or Nothing (i.e. surf the interwebs). And it's with this decision that every reasonably happy, veteran developer I know distinguishes themselves. They all choose Extra.

But aren't More and Extra the same thing? No! They sound similar, but they're actually very different. Here's a simple example:

Say for this sprint you're assigned two form screens that take user input and then persist it to the database. Pretty straight-forward. This is your Normal Work - the default expectation of what you need to get done to be in good standing on your project.

Doing More would be completing those two screens and then taking on a third screen that's just like it. Yes, this would help move the project along faster and make your manager happy, and that's great, but in the long-run, More doesn't give you much.

Extra is different than More. Extra is finishing those two screens, but then researching a new library for form validation that might reduce the boilerplate code. Or it's learning ways to protect against common security vulnerabilities from data entry. These little off-ramps from the main highway of Normal Work could be dead-ends and not have any practical value to the project. But they might also be important contributions. And that's the thing with Extra. While the tangible value to the project is uncertain (it could be nothing this time or it could be something), the value to you is real.

Extra gives back in a couple ways. The first is philosophical. Extra is about having free will. It's about not being just a drone who monotonously does exactly what their told, but someone who asserts a little control and shapes their our own contributions and experiences in a way that's interesting or meaningful.

More practically though, Extra is also good for our careers. Our employability has everything to do with the diversity and depth of our experiences, and so Extra is what helps us broaden ourselves beyond just what's useful on our narrow project. Whereas grinding it out to finish that third screen wouldn't add any new skill or insight to our toolset, a little bit of exploration from Extra could.

Now there are of course caveats. Over the years I've seen developers get carried away with Extra. Extra is all they want to do. They spend their days researching new validation frameworks or security vulnerabilities, and then can't get their Normal Work done, those two basic screens. And sure this might be more fun for them, but it sucks for the rest of us because we have to pick up their slack if we want to deliver the project on time. So there are rules to Extra.

The first rule is this: Extra must be balanced against Normal Work. Real stuff still needs to get done. This is how we get paid. Shirking our Normal Work in favor of Extra might be more interesting, but it makes us shitty teammates at best, and unethical at worst. Note: there are a few "lucky" among us who find their way into roles where Extra is their Normal Work, and we call these people Architects. ;)

The second rule of Extra is that it must be aligned with your Normal Work. If what's on your plate is two form screens, Extra can't be learning about evolutionary algorithms or building a mobile app for your side hustle. It's got to be something that could help the project. It's never something we need to hide, but instead it's something we're eager to share with our teams - ala "hey, I did some research on X, and maybe this is something that could be valuable for us to try".

In the end, I hope it's clear that I'm not arguing that we neglect our Normal Work. This is what is expected of us, but the choice is what we do beyond this: Nothing, More, or Extra. In my experience, those who have had longevity in this field always choose Extra. I'd love to hear your thoughts.

Of course they're not as central to the process as they were years ago, but some companies still use them to help decide who to interview, and some interviewers still use them as a guide for what questions to ask. And in either of these scenarios, if you're the one who has to read through the resume, it's not a great experience.

The problem is our typical developer resume is a multi-page chronicle of every system, component, and library we've ever worked with over the course of our entire careers. Information overload. If the reader had only a few minutes to skim our resume (which is usually the case), it'd be really hard for them to come away with an accurate picture of what we know or what we've done. They might wonder, "how much has he worked with single page apps?", and then would scan the resume like a grade school word search until their few minutes were up.

But it makes sense why it's this way. From a developer's perspective, it behooves us to cram in as much information as possible, lest we miss out on conveying some skill we've honed or technology we've used that might be relevant to some company. It's like packing for a month-long vacation where you don't know whether you'll be skiing, sunbathing, or going to a formal dinner. Might as well pack it all! And so our resumes become absurdly long.

My goal with this project was to improve on that user experience, even if just a little. Make the resume more of an interactive dashboard than a static document - one that allows us to keep all the detail, but lets the reader zoom in and out to view what's most interesting. So given this guiding vision, I built two main features for the MVP:

First, there's a timeline view at the top, so the resume reader can quickly get the "big picture" of your career trajectory. How long have you been different places? Have you been going to school and working at the same time? How long have you been a "senior"? In my opinion, seeing this visually makes this way easier.

The second feature is a full-resume skill filter. Each experience is tagged by skill, and then these skills can be used to filter the resume content. So if you want to see how much I've worked with NodeJS, then clicking on "NodeJS" would filter the entire resume down to only those experiences where I used it.

There are a few other bells and whistles that hopefully make this a viable way to maintain and share your resume, like PDF download and custom themes, but that's the gist of it as of now.

I have no idea whether this will fly, but it's been fun working on it. Give it a try at TechRez.io. And if you have any feedback or ideas, I'd love to hear it either in the comments below, or you can send feedback on the site.

We're ready to start our journey. We have an application that's riddled with technical debt, woefully out of date, or just generally underserving its users, and so we need to understand what our best option is going forward - does it make more sense to continue to plod along and incrementally refactor? Or do we blow it all up and rewrite from scratch? This is the basic dilemma we'll be exploring in this book. So let's get going...

But not so fast! Before we go any further we need to address the elephant in the room, which is this: for any legacy application in need of improvement, what to do next is not a simple this-or-that decision. We may frame our options generally as rewrite or refactor, but these terms, as we'll see, are really just stand-ins for a whole spectrum of choices that lie before us.

By refactoring, within the context of modernizing a legacy application, we typically mean that we're going to keep the application mostly as it is, but make some minor, internal improvements to solve specific problems (like maintainability, extensibility, etc.). By rewriting, on the other hand, we imply that we intend to "start again from scratch", or in other words to make major changes.

But this only begs the next question! What exactly do we mean by minor and major? If we plan to upgrade our frontend framework from AngularJS to React but keep the backend services as they are, is that a refactor or rewrite? Or what about if we want to break a monolith into three different microservices, but the business logic is just copied-and-pasted into new version-control repos - is that a rewrite, refactor, or something else? And do we even care? Does labeling our effort really matter?

Yes, it does. Although our job is to build working software and not philosophize about semantics, the words we use do make a difference. When we suggest the path of rewriting or to refactoring, business and technology stakeholders should understand exactly what we mean and what type of effort will be entailed. In other words, some precision in our terms will help us better set expectations. Further, as we burn through some of this conceptual fog and find clearer definitions, it will also give us a more nuanced view of this decision, and move us beyond the narrow rewrite or refactor framing.

So like any big journey, let's spend a little time packing before we jump in the car and go. We don't want to show up at the beach and realize we forgot our swim suit.

Functional Improvement

A good place to start is to define what rewrites and refactors are not, which are strategies to improve the functionality of an application. This type of work, whether it's fixing defects, delivering new features, or cleaning up the user interface, we can call enhancement. It's about improving on what the application does for its users, and as we'll see later, it's the normal state of development.

In some cases, the scope of the functional enhancements can be quite large though. The business may determine that the app serves the right user base, for example, but the features all need to be overhauled. It may be tempting to call this case a rewrite as well, however we're going to make a distinction here. Because this type of effort entails building all new functionality, it is basically indistinguishable from greenfield development. When new functional requirements need to be defined, a separate system is developed from scratch, and no carry over of logic or code is possible, we will consider this to be development of a greenfield app and not a rewrite.

What we mean by Refactoring

Adding functionality to an application is not what this book is about. Our situation is that the application generally does what it is expected to do, but is lacking in the how - in other words, the non-functional or quality attributes of the system. For example, users might be happy with the set of features, but the application might be excessively difficult to maintain, or it may crash frequently, or may perform poorly under peak load. It's when these non-functional attributes are lacking that we consider a rewrite or refactor.

With respect to refactoring, we often use this term to refer to different scopes of work. In his book Refactoring, Martin Fowler defines it this way:

Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.

In this purist sense, refactoring is primarily about making the code more maintainable. This might be decomposing long or complex functions, fixing inconsistent naming, adding unit tests, or restructuring class hierarchies, data structures, or schemas. Note that nothing that is visible to the user changes, but the internal code structure is modified to make it easier to work with for developers, thus improving our productivity (and happiness!).

In the context of our rewrite or refactor decision, however, this definition is too restrictive. When we talk about refactoring in this context, we're not typically making a distinction between internal and external, but rather about functional and non-functional. For example, we may say that we are choosing to refactor the existing code base to improve the reliability or performance of the application. These non-functional attributes are technically not internal properties of the system (they are very visible to the users in that they directly impact them), they are just non-functional. This might be a pedantic distinction, but in the spirit of precision, I think it's important to call out. In this book we'll be using a broader definition of refactoring:

Refactoring is an approach by which an existing body of code is incrementally restructured in order to improve the non-functional attributes of the system.

Lastly, it's important to note that refactoring is about iterative change. It's making minor tweaks to the application, delivering them, and then rinse and repeat. Layered in with functional enhancements, refactoring can keep our users happy and our codes base healthy, minimizing technical debt and feature gaps. When neglected, however, we may need to consider the more heavy-weight alternative.

What we mean by Rewriting

Like refactoring, rewriting has the same basic goal - to improve the non-functional nature of the application. The difference is in how much is changed. Simply put, if refactoring is duct tape, rewriting is a sledge hammer or a back hoe. It's not about making incremental improvements to what is there, it's about blowing it up and building anew. With respect to the other types of development efforts we've discussed, we can visualize a rewrite this way:

We can say then that a rewrite is an effort that involves major changes to the system in order to make fundamental improvements to its non-functional attributes. But there is gray area. Rewrite efforts often spill over into the other quadrants. For example, an application may be so crippled by technical debt that it's virtually impossible to add new features. We may choose then to rewrite and build a new foundation to improve maintainability and extensibility (non-functional attributes), but during the course of that rewrite we may also sneak in a few new features to satisfy the business. It's fundamentally a rewrite, but there is some enhancement going on as well.

Likewise, there is some fuzziness at the boundary of rewrite and refactor. There are lift-and-shift scenarios where the system is moved to a new platform making it essentially a different application, but the code implementation within it is mostly the same - i.e. not refactored. This feels like a rewrite, but is it? How much do we have to change to consider it a rewrite?

Again, let's see if we can add some precision. For the purpose of this book, let's use the following definition:

Rewriting is re-building the same functionality that exists in a legacy application but using a different language/framework, maintaining within a new code repository (not just a branch), and deploying as an entirely new artifact (possibly to a different platform - e.g. servers, hardware, serverless, client, etc.).

In other words, we're drawing some clear lines. If, for example, we're rewriting an important function, class, or even module, but our work is made in a branch off of the mainline of our codebase, it's not a rewrite. Likewise, if we re-implement a segment of the application, but the system itself is still deployed as the same artifact (binary, WAR, etc.), this also is not a rewrite. Rewrites within our context are BIG. They are about major changes that necessitate an entirely new application to be built and deployed. Yes, there may be incremental steps along the road to get there, as we'll see later, but it is a fundamentally different effort than refactoring.

To help clarify things in your individual case, it can be helpful to diagram this out. For the different possible paths for modernizing or improving your application, what exactly is being changed? Here's an example:

In practice, the nature of the changes may not line up neatly with the definitions of rewrite or refactor, and that's ok. The diagram above, for example, might represent a case where we're proposing to re-implement a service using a more modern set of technologies, but while keeping the exposed API and underlying persistence structure the same. This represents a blend of minor and major changes, so it still may not be clear precisely what to label it. What's important, however, is that we've arrived at deeper level of detail which will help us better reason about and justify the decision.

Now we're almost done with our preparations, but before we get on with our journey, let's put it all together and see how these different types of development efforts all fit within the lifecycle of a given application. In other words, let's explore the origin story for a rewrite.

The Rewrite Origin Story

It all starts in the Greenfield Development stage, where we take an idea and begin to build from it a functioning application. After weeks or months of dogged effort, something is eventually Launched to "market" (which may be actual paying customers, or just a set of internal business users, etc.). If the app is well received, it lives in the equilibrium stage of Enhancement for some period of time, where new features are added and defects are resolved. Everyone is happy. Eventually though, technical debt can accumulate, and we begin to see Diminishing Returns in our effort - whereas a week of development used to be enough to add an entire new feature, it might barely suffice now to change the color of a button.

It's at this point that we might question whether it's worth investing more time and money. Further, since the app was first launched, some exciting new technologies have probably emerged, and we may have some grand visions for how they could be leveraged to make our app more resilient, easier to work with, performant, etc. And so we begin to make a plan for a rewrite. The idea is to freeze development on the existing system for some short period of time, and then shift resources over to work on the replacement system. We'll first build the foundation (using more modern patterns, tools, languages, etc.) and then next migrate the existing functionality into it. Users will just need to ride out the "pause" (i.e. not get any new updates), but when the rewritten system is in place, productivity should be twofold (or more!) what it was before.

And while this plan seem may seem straight forward, it belies some critical risks - technical, organizational, and psychological factors that all conspire to make this rewrite stage extremely precarious. And as this phase drags out, our chances making it out with a successful replacement get dimmer and dimmer. In the next two chapters we'll explore some of these dangers lurking within a rewrite effort, and then the reasons why we often forge ahead anyway in spite of them.

The system is already in production, we obviously know how it works, so we just need to port what we have to a better platform, and once we're there things will be easier.

In this chapter, we're going to rip this basic intuition to shreds. Rewrites, as we'll see, are anything but easy. While they may be immune to some of the challenges of a nascent app, they introduce completely new ones we often don't anticipate. In order to successfully deliver a rewrite, we must navigate these challenges, and so it's important to understand what they are up front.

Parallel Development

The first hazard we often confront on our journey is the realization that a rewrite is fundamentally a dual prong effort. Sure, on the technology side we'd love to think we can pause development to focus only on writing code in the new platform, but reality (and more specifically, the business) rarely abides. Maybe in order to win (or keep) an important customer, the business demands we add some new feature to the existing system NOW. Or maybe a third-party system has changed its API and we need to refactor, or a sev-1 defect pops up, or a new government regulation is written. The point is, almost never can a production system sit static in space and time for long, and that's true even during a rewrite period. Some care and maintenance will inevitably be necessary on the old system, which means parallel development.

The first problem here is that developing in parallel violates that principle we developers hold most sacred: Don't Repeat Yourself. Any change made to the legacy system will need to be ported into the new system as well. For example, if we add feature X into the old stack, we must add X again in the new one (but this time write it in the new language or framework). And that duplicative effort applies for the testing, project management, build, and deployment of X as well. Essentially, whatever it would cost (in time or resources) to make a change to the legacy system, it could now, during this rewrite phase, easily cost double.

Even more, depending on how far along the rewrite effort is, the change (e.g. feature, defect fix, etc.) might need to be made to a part of the source code that has not yet been rewritten, and so the team will have to remember and account for it when they get there (which will then push the timeline back for the rewrite). Or in the opposite case, the change is in a section of the app that has already been rewritten, and so the team has to circle back and make updates to code that was just migrated (and then re-test, etc.). Either way, the parallel development element of a rewrite can make it feel like the team is keeping a plane flying while also building its replacement on the ground. For the safety of the passengers, that plane's got to stay in the air, but if we spend too much focusing on its maintenance, we'll never get the new one off the runway. And of course the team responsible for both "planes" is often one and the same, which leads to the next issue...

Team Schism

Given that there's some amount of effort necessary to support the existing system, the question is: who is responsible for this? There are a couple approaches that teams typically take to manage the parallel work. One option is to have the more junior developers stay back in maintenance mode, and thereby free up the senior devs to forge the path forward for the new system. And this makes sense. To pull off a rewrite generally requires a higher level of technical competence - getting new technologies up and running and working together, configuring environments, establishing patterns and conventions, and so on are all tricky tasks that could leave a more green developer spinning their wheels in the mud.

And so teams may settle on this type of split, junior developers to maintain, senior developers to go forward. And it may work for a short while, but it's not long until the senior devs are pulled back in to help solve a critical all-hands-on-deck issue or discuss the impact of some change. Their years of acquired knowledge of the old system and their relationships with business stakeholders, ops people, and others make them effectively irreplaceable. They need to be included. And even if these "pull-outs" only take up a couple hours a week, that level of context-switching can still stymie productivity. As developers float back and forth between the old and new systems, the timeline for the rewrite just gets pushed back.

To avoid this, teams may take an alternative approach: spin up a new team that can be more easily dedicated to the rewrite. The developers on this "rewrite team" would have negligible ties to the existing system (maybe they're consultants, new hires, etc.) and would therefore be better able to isolate themselves and stay productive on the new work. Problem solved? No. The first issue, of course, is that these developers are by definition less familiar with the codebase, domain, and generally how and why things work the way they do. Sure, they have the original code as documentation, but this can be like learning a programming language by looking at the source. It's all facts, but no story. And so this new team will either flail while trying to understand the intricacies and complexities of the existing system, or make bad assumptions just to move forward, or end up having to pull in the original team anyway. None of these are productive outcomes.

Additionally, this "fresh team" approach can also alienate the veteran developers who are charged with staying back to support the legacy system, and make them feel as though they are being penalized for their loyalty and experience. While the new team gets to work with modern technologies and a clean slate, they are still left in the doldrums of maintenance. No fun. And so unsurprisingly, this can quickly lead to disgruntled developers and shortly after that, brain-drain. Somehow a team structure needs to be devised where the old system can be supported, the new system can be built correctly and efficiently, and everyone is happy. But even when you put the right people on the right busses, there are still surprises...

Unknown Unknowns

Now even if the rewrite team is staffed with seasoned developers - i.e. those who either helped write the legacy system or at least have some experience with it - there are always surprises when migrating the existing source code to the new platform. Donald Rumsfeld (yes, that Donald Rumsfeld) made this not-so-crazy insight about these types of surprises:

Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.

When we begin the rewrite process, we typically try to estimate how much effort it will be. And it's here where we begin to catalogue the first two categories of work. There are of course segments of the codebase that we have direct experience with, and so we give quicker (and more accurate) estimates. These are the known knowns. But then there are also pockets of the system that we recognize we have less direct experience with, and so we add a little buffer. "Joe wrote the registration flow, and he quit last year, so that might take a little longer to migrate". These are the known unknowns. And so we proceed with planning, estimating both types of work and giving extra time for the known unknowns, until an overall timeline is agreed to.

It's not until we dig in and start migrating code, however, that we stumble upon that third and most pernicious category of work, the unknown unknowns. It may be a 1,000 lines of convoluted business logic we never knew existed, or a set of reports we didn't know were used by anyone, or some "duct tape" that unbeknownst to us holds an entire integration together. Whatever it is, our original rewrite plan never took it into account, but we're in too far at this point to bail out. The result is that this (formerly) unknown unknown will need to be be analyzed ("what the heck does it do?") and then dealt with ("do we need to bring this over, or can we leave it behind?"). And this extra analysis, discussion, and effort just pushes back the initial timeline and strains the patience of business sponsors and customers.

And while a few of these unknown unknowns can be absorbed, too many can put the entire rewrite effort in jeopardy. With better planning and decomposition (more on this later), they can be minimized, but avoiding them altogether is difficult. There is another hazard, however, that we inflict on ourselves...

The Second System Effect

Often, we've been living with the warts and imperfections of the existing system for so long that when we finally get the chance for a do-over, we can't help but want to make everything better. Or perfect. Decades ago, Fred Brooks labeled this tendency the Second System Effect. In the Mythical Man Month, speaking of the architect of the system, Brooks observes...

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

This effect can manifest itself in a few ways. First, we tend to see the rewrite as an opportunity to eliminate all the technical debt we accrued in the past system. We want to decompose God classes, fix inconsistent variable naming, re-shuffle and restructure data structures, and so on. Basically, we see this as our chance to resolve all of our little peccadillos with the legacy code, because it wouldn't feel right to port that debt into the new codebase.

Additionally, it's also easy to think of the rewrite as not just a chance to modernize certain pieces of the architecture that are flawed, deficient, or out of support, but also as a vehicle to leap-frog everything to the cutting edge. Maybe the UI is still the old AngularJS and doesn't render well for mobile web users, so it does seem justified to port to a more modern web framework. But while we're at it, let's also break the backend up into microservices and write them in Go! Essentially, the rewrite can become like a piece of legislation - it may start out as a simple bill to reduce gun violence, but by the time it gets passed into law West Virginia is getting five new bridges and Oregon's farmers are being subsidized to grow soy.

And the Second System Effect is not just specific to developers. Oftentimes business stakeholders will take the same tact, but with their priorities. "We might as well add this feature to the rewrite that the customers have been dying for, since we'll be touching that code anyway." And just as with the technical improvements and embellishments, the scope just gets bigger and bigger and the release date of the rewrite gets pushed back further and further. In the end, while the mantra of development for the initial system was "get something up and running fast!", for the rewrite it can be "we'll never get another chance to do X, Y, and Z!"

Fighting the Last War

It's often said that generals in peace time prepare themselves to fight the last war. They look back and reflect on the flaws and faults of their old battle plans, and vow to never let those mistakes happen again. And then the next war comes, and it's nothing like the last. Their preparation was for naught. The same can happen with a rewrite. We are so keenly aware of our mis-steps of the first system, that when it's time to design the second, we make sure above all else that we don't get burned again in the same way. But it's easy to miss that things have changed. Those past mistakes may be non-issues at the point we rewrite, and so it's not necessary to go to any ends to prevent them. At the same time the modern technologies being used in the rewrite have introduced a whole new set of issues that we haven't yet looked ahead to spot.

In my early years of consulting, I helped build for a client a large web-app that served a few thousand internal users. The application was a success - meaning that we delivered the functionality within time and budget (plus or minus), but we unknowingly hitched our "horse" to the wrong wagon, so to speak. These were the early days of single-page apps, so we used a budding framework called Google Web Toolkit, which was quite cool at the time (you'll have to take my word on this). Unfortunately, a year or so after this app was launched, Google effectively jumped ship from GWT in favor of something better (Angular), leaving everyone (including this client) stranded on an essentially unsupported technology. Not a good situation, though in our defense it would have been difficult to predict.

Years later, I had the chance to reconnect with some of the developers still there, and they told me they had no choice but to rewrite the app. And rightly so. But when I asked what tech stack they had picked, I was surprised. Not wanting to be burned by single-page app technology or a flash-in-the-pan framework, they chose an incredibly mature but decade-old server-side page rendering technology, despite the fact that single-page frameworks were, at that point, extremely solid and used by, well, everyone, and their app had some very rich user interaction that could have benefitted from that model. In other words, the central driver for their rewrite was to avoid the mistakes of the first system, but in the process they missed out on many of the advantages of modern web frameworks. They were fighting the last war.

All or Nothing

The last big hazard of a rewrite is something that can be avoided (as we'll discuss in later chapters), but often is not. While we now accept the wisdom of iterative development and the concept of the Minimum Viable Product, with a rewrite it can be difficult to find a way around delivering everything all at once in one big bang. For example, if the customers have been using the legacy system which has, say, one hundred features, how can we deliver a new system with anything less than those one hundred features if we expect them to (happily) switch over? Basically, the rewrite needs to do everything the old system did, and so it would seem that we must first deploy the entire replacement before we can get back to iterative development. Depending on how big the existing system is though, this can represent a considerable effort.

Further, this "all or nothing" requirement can combine with the ever constant demands of the business to put incredible pressure on the team. If the team can't deliver anything until it's all finished, then they aren't able to show any tangible value until the replacement is launched. The business is in a sense being held captive by the rewrite effort, and it's up to them to stay patient and trust in their development team to finish. Not an easy feat. If the customers are especially demanding or if the team consistently hits unexpected delays and issues (e.g. "unknown unknowns", etc.) that push the release date out for the rewrite, the business might feel like it has to apply pressure (which can force the team to cut corners) or can pull the plug altogether.

Summing Up

In the end, hopefully some of the dangers that lurk in the Rewrite phase are now more visible. While the greenfield stage of development surely has its perils, the rewrite is no walk in the park either. The challenges of parallel development, team organization, feature and technology gold plating, and big-bang deployment can easily drag down the most competent of teams. And so it would seem that we would be apprehensive about taking on a massive rewrite. As we'll see in the next chapter, surprisingly, this isn't always the case. Despite the mortal dangers to teams in the rewrite phase, we often put caution to the wind and take the journey of a rewrite anyway.

As we saw in the previous chapter, a software rewrite can be a dangerous undertaking - there are a number of very real costs and risks that can sink even the most well-intentioned effort. And yet we do rewrite! Often. Despite the warnings and obvious dangers, we come together, business and technology, and collectively declare: "damn the torpedoes, we're rewriting this old pile of code, and this time we'll succeed!"

The question I'd like to explore in this chapter is "why". Given all the costs and risks, why do we so often choose to rewrite a working legacy system rather than refactor it in place? Leaning on insights from psychology, sociology, and other fields, I'd like to expose some subtle and not-so subtle forces that serve to nudge each of us toward the path of the full-scale rewrite. Though none of these individually are the ultimate reason we choose "big modernization", together I think they form a powerful undercurrent that float us in that direction.

I want to be clear though. I'm absolutely not arguing that a rewrite is always the wrong choice - there are many good and justified reasons to rewrite, which we'll see in the next chapter. And further I'm not trying to cast aspersions on developers or other stakeholders for making "bad" decisions. We do all have blind spots and biases, however. When we're aware of them, we can work to counteract them - which in this case might mean sparing ourselves and our organizations from sinking countless hours and dollars into what might be a doomed rewrite. With those caveats out of the way, let's take a look at some of these forces at play...

The Enjoyment of Creation

Developers enjoy the act of creation. We are builders by nature. Though we understand that the job often requires us to bravely venture into the dark recesses of others' code to slay some lurking defect or tweak some abstruse calculation, this isn't what attracts us to programming. It's just a necessary evil. When given the chance, we want to build our own systems. We want to craft a solution to a problem in the way that makes sense to us, in the style that appeals to us. At the risk of sentimentalizing, programming is our art. In the same way a painter would prefer to start from a clean canvas than paint within the portrait of another, all things being equal, we too want to give life to our own creation.

In fact, there is evidence that this fondness we have for the things we've made ourselves is not just a part of the psyche of programmers but is built into all of us. Dan Ariely, in his book Payoff: The Hidden Logic That Shapes Our Motivations, gives an interesting anecdote that underscores this human tendency:

Consider cake mixes. Back in the 1940s, when most women worked at home, a company called P. Duff and Sons introduced boxed cake mixes. Housewives had only to add water, stir up the batter in a bowl, pour it into a cake pan, bake it for half an hour, and voila ! They had a tasty dessert. But surprisingly, these mixes didn't sell well. The reason had nothing to do with the taste. It had to do with the complexity of the process but not in the way we usually think about complexity.

Duff discovered that the housewives felt these cakes did not feel like the housewives' own creations; there was simply too little effort involved to confer a sense of creation and meaningful ownership. So the company took the eggs and milk powder out of the mix. This time, when the housewives added fresh eggs, oil, and real milk, they felt like they'd participated in the making and were much happier with the end product.

Ariely goes on to describe a number of scientific studies that show this exact same effect: we are in fact hard-wired to enjoy creating things, and further when we do create, we overly value that which we've built to the creations of others (independent of how good what we've produced actually is!).

I think this directly speaks to how we as developers can feel about maintaining legacy systems. Sure, we're capable of rotely following some prescribed pattern to add a new column to the report or button to the screen, but that doesn't give us that feeling of attachment or meaning to our work. What we'd like is to crack our own eggs and mix in our own milk and oil. In other words, our brains may naturally tend toward wanting to rewrite over refactor.

The Drudgery of Maintenance

The flip side of seeking out enjoyment from the act of creation is avoiding frustration from the drudgery of maintenance. And there is plenty of frustration built into legacy systems. Over time, software tends toward decay. Business requirements shift, forcing developers to remodel what used to be the kitchen into the front porch. Critical deadlines force us to cut corners, and ship code with TODOs and copy-and-paste. And the comings-and-goings of old and new developers to the system create what, over time, look like sedimentary layers in the architecture - "you can see from the style that this code comes from the pre-IPO period, before the time of dependency injection".

Working within legacy code like this can be frustrating to say the least. To fix even the simplest bug might require us to trace through convoluted execution paths as they wind in and out of different software ghettos, where weeds have grown tall and broken windows have gone un-fixed. Keeping the logic (or illogic!) of the system in our heads can be a herculean task. And while we dutifully accept our fate and fix those defects and add those small enhancements, all the while we think about how we wish we could do it differently. We dream about how cathartic it would be to put a torch to this bastion of complexity and technical debt and start over from scratch.

To the end user and perhaps even to the business, however, it may be a different story. Sure the legacy system might not be as visually pleasing or it might require a few more clicks or seconds than might be preferred, but it likely still works. The system is functional, relatively reliable, and the business processes around it, though not optimal, are at least well understood. But to the developer, especially one who was not the original creator (see above!), the system can represent not comfort but frustration and constraint. And the more time we work within it, the more we want to escape and rewrite it.

Self-Interest and Fashion

Beyond just seeking enjoyment or avoiding frustration, we as developers are also well aware of what tangibly benefits us and our careers, and often it's not maintaining legacy systems. A 2019 Stack Overflow developer survey shows a direct correlation between the newness of a language and the amount of money a developer makes. Whereas the average age of top 5 languages by salary is just 15 years old, the average age of the bottom 5 languages is 39 years old.

Top 5 Languages by Average Salary

Bottom 5 Languages by Average Salary

Now of course many other factors play into these numbers, but all things being equal, it's pretty clear that if we don't keep pace with newer technologies, the amount of money we can expect to make will be less. This is probably not surprising for any developer who has recently been on the job market. Recruiters, resume bots, and even fellow developers are hunting through our experiences for the latest buzz words: microservices, Kubernetes, progressive web apps, or whatever is the technology du jour. The newer and more cutting edge, the better. The logic, presumably, is that a developer who is up on the latest skills must be more motivated and invested in their craft than someone who has only been doing the yeoman's work of maintaining a legacy system, (the irony being that working within the legacy system might have been the right thing from the perspective of the prior employer!).

Now some of this does make sense. The job of programming does entail a high degree of exploration and experimentation, and so a passion for learning is indeed important. However, there's a tendency, I think, for our industry to sometimes favor fashion over substance. When we pick a language, tool, or framework, of course we want to choose the one that is right for the job, but we also consider what the choice says about us. Are we the kind of person that uses a proven but stodgy language like Java, or are we "forward-thinking" and "innovative" and pick Go or Kotlin? In a sense, the technologies we use are like the clothes we wear - they speak to who we are, what we stand for, our worth. It can be technology as fashion.

Now of course the analogy is far from perfect, but consider some parallels from this description of the fashion cycle by sociologist Herbert Blumer:

The elite class seeks to set itself apart by observable marks or insignia, such as distinctive forms of dress. However, members of the immediately subjacent classes adopt these insignia as a means of satisfying their striving to identify with a superior status. They, in turn, are copied by members of classes beneath them.

In this way, the distinguishing insignia of the elite class filter down through the class pyramid. In this process, however, the elite class loses these marks of separate identity. It is led, accordingly, to devise new distinguishing insignia which, again, are copied by the classes below, thus repeating the cycle.

In a nutshell, fashion, whether in clothing, music, or programming, is about class differentiation. The elites innovate. The masses catch up so they can be associated with the elites. As soon as they do, however, the elites are no longer "elite", and so they must break away and innovate again. The masses eventually follow, and the cycle continues, ad inifinitum.

When it comes to the question of rewriting or refactoring, I think this can be another subtle force that pushes us as developers toward the choice of full-on modernization. To keep an existing system alive is akin to wearing your clothes from a decade ago. Sure, they may still keep you warm and dry, but they are old and worn and probably don't present to the world the image you want show. And for those who aren't prone to the frivolousness of fashion, we have to at least know that it does tangibly affect our careers. Keep pace or make less.

Faulty Intuition and Mental Short-cuts

Putting motivations and self-interest aside, there is another powerful force which can compel us to believe that rewriting is the obvious and best decision for our legacy systems even when it's not: our own intuition. In his amazing book on decision making and cognitive bias, Thinking Fast and Slow, Daniel Kahneman explains that while it's easy for us to form opinions on all types of questions (and to be confident in these opinions to boot!) oftentimes they're based on simple mental short-cuts, and not solid reasoning. For decisions where the stakes are small this might be ok, but for the question of whether to refactor or rewrite an entire system, where hundreds if not thousands of hours of effort could be at stake, this isn't going to cut it. We need to be sure.

One of these subtle mental short-cuts we use is called substitution. When confronted with a question that would require deep effort and analysis (which is hard work!), we often just swap in an easier proxy question that we can answer more quickly. For example, to know whether it's worth it to rewrite vs. refactor, we should understand what the cost of a rewrite would be, how much value it would bring to the organization, when ROI would be realized, and so on. But this is hard! So instead, we substitute in questions that are easier to answer like "is the existing system buggy?" or "is the UI out-of-date?". You bet it is! And while it seems like we addressed the original (hard) question, we actually didn't - we just swapped it for something simpler.

Over the years, I've seen this type of justification over and over. Rather than pause to try to answer the real but difficult question - whether the net value of a rewrite is greater than the net value of a refactor - we often fall back to easier substitute questions. We say things like "we're going to rewrite because the legacy system is hard to work with" or "because no one understands it". Yes, these answers are relevant, but they're just not sufficient. Maybe it's true that no one understands the code, but how much would it cost for a person to gain that understanding? And is that cost greater than or less than rewriting the entire system from scratch (which would probably require understanding it all anyway!). Ditto for bugs or tech debt. What are the costs and returns?

Now of course we live in a world that moves fast - we don't have infinite time to analyze, make business cases, and pontificate. But for a question as crucial as rewrite or refactor, we should try to slow down a little and make sure we're at least answering the real question, and not an easier substitute one.

Throw-Away Culture

This last force that can push us toward a rewrite is not rooted within us, but rather in the society around us. We live in a throw-away culture. Our food and beverages come in one-time use containers. The products we buy are garbed in excessive plastic and paper packaging. And our appliances and devices are designed for obsolescence, so they can't be repaired even if we wanted to. In other words, we've been conditioned to throw things away without a second thought - whether its dented, cracked, malfunctioning, or just slightly old or worn, we toss it out.

And so, I think it's at least possible that this same "throw-away first" attitude affects the way we think about our software as well. When a system ages, our inclination is not to pull out the duct tape and WD40 to give it a few more years of life, it's to throw it to the curb and go out and get (or in this case build!) the newer model. In fact, it seems that the very word "legacy" can induce groans of sympathy from our colleagues. "I'm sorry you have to work on that", they say, "when are you going to rewrite?"

Now to be fair, our industry is one of constant improvement and innovation. We are continually learning better and cleaner ways of doing things and then incorporating them into our tools, frameworks, and languages. No one would ever purposely use Java 7 when they could use streams and lambdas from Java 8, or even a more concise or functional language like Kotlin or Scala. Additionally, new platforms and devices are constantly being released, which our existing systems may not port to. So I'm not saying that scrapping the old for the new is necessarily a frivolous choice, as we'll see in the next chapter.

I do think it's helpful though, like with the other forces, to be aware of this potential bias. In the same way that there are very real costs to our society for our thoughtless waste, there can be costs to the organization when we too quickly abandon a legacy system that may be reliable, albeit a little old or complex. In the following chapters, I'd like to argue for a different three "R"s of legacy modernization: "repair", "reuse", and "recycle". Instead of seeing our existing systems as things we need to distance ourselves from, I think we should appreciate and leverage the components and elements that are working, and only seek to rewrite that which we need to.

Wrapping Up

At this point, we've seen the many dangers and risks that await us on our journey to rewrite, and we've seen the subtle forces that nudge us in that direction despite it all. In the next chapter we'll look at when a full-scale rewrite does make sense - the reasons, constraints, and drivers that justifiably push us to abandon our existing systems and start afresh. If done for the right reasons, a rewrite can be a real opportunity to improve things we care about - user experience, maintainability, performance, and the ability to add new features and address new use cases.

There's this tough problem the developers have been struggling with for weeks (performance, or scalability, or whatever), so can you quickly look at it and show them the best practice, quick fix, etc. that solves it all?

Reasonable ask? No, not really. The expectation is that the architect is a chess master. It's just like that movie we've all seen. There's the young or disheveled or [insert surprising adjective] savant who, walking through central park mid-dialogue with someone else, glances for a nano-second at a pair playing chess and declares "bishop to queen 4, check mate" and then keeps walking. Minds blown.

But why shouldn't an architect be able to size things up as quickly as a chess master? Both an architect and a chess master have accumulated thousands of hours of experience in their disciplines. Both are analyzing current states of a "board" and then evaluating the trade-offs of different next moves or options.

The difference is that chess masters are always playing the exact same game. Rooks always go forward-backward and side-to-side. Bishops always move on diagonals. And because the rules don't ever change, it's easier to take a mental snapshot of a board and instantly pattern match back to some other past experience - e.g. "this is exactly like the game I played/studied four months ago".

Any given software system, on the other hand, may look the same as countless others you've worked on in the past - I mean, hey, it's all ifs and elses, classes and objects, REST and SOAP calls, etc. - but just a little bit below the surface we realize the pieces and the rules are very different. In my system the rook can jump the first 2 squares but in yours the queen has 3 lives. Each system we visit has different intricacies and rules we must learn anew.

This immense variability and complexity is baked into even the simplest software system - it's in the business domain, in the frameworks/techologies used, and in the implementation itself - and so it takes time to grok all these nuances and details before we can build the right mental picture that allows us to reason about it effectively. Even for those with years of experience this is true. Sure there are common patterns, styles, tools, specifications, etc. that, if you know them, can help in terms of more quickly sizing up a system, but almost never do these alone allow you to pull off that type of chess savant "mate in one" mastery. Real-world software solutions take time. Architecture takes time.

This is why I think it's important to think about software architecture as a process and not a destination. Being an architect (or playing the role of an architect) doesn't mean simply divining the "right" solution or recommendation and handing it off to the developers. Anyone can whip up a bunch of boxes and lines that seem plausible, but do they really apply to the game of "chess" you're playing now, or the one from your past project or from that popular blog post? To truly know the right next "move" requires first understanding the rules of the game you're in. What are the business drivers, risks, technical constraints, legacy components, integrations, possible options, trade-offs, and so on. These are the details that take time to identify, clarify, and understand, and they are subtly (and sometimes not so subtly) different from project to project and organization to organization.

In my opinion, the expectation for the architect should be to setup the framework or process by which good architectural "moves" are made (Michael Keeling describes this well in Design It). Thinking an architect is going to stroll into a meeting and find a check mate lurking in plain site is sadly unrealistic.

You're an avid marathon runner, and are pretty fast to boot. It's your first race as part of a new running club, and you're excited. The club is big and, some say, impersonal, but you were lured by its longevity and reputation...and of course the kick-ass benefits - massages, gear, the works.

Anyway, it's the morning of the race, and you show up at the course. You head over to the club's meeting spot, and the first person who spots you is the health specialist. "Nice that they care so much about my health to have a dedicated specialist", you think.

After a friendly introduction, the health specialist says, "We need you to wear this backpack while you run, for your safety".

"Seriously?" you ask. "Won't it slow me down?"

"Well, it might, but it's really important because it has all types of essentials in it: bandaids, wraps, braces, ice packs, and a book of medical conditions. All our runners wear them, so it shouldn't be a big deal."

"Ok", you say reluctantly, "if you say so", and you strap on your backpack. The thing must way 15 pounds.

At that moment, the club's race coordinator ambles over. "Hey-o", he greets you warmly. "It's really important to know where all our runners are during the race, so we'd like you to strap on this ankle bracelet tracking device".

Again, you push back a little, "Won't this rub on my ankle and give me a blister? And can't I just use my phone?"

"Nope, I'm sorry", he says, "club policy."

As you finish attaching the bracelet to your ankle, you look up, and are met by the club's gear specialist. She hands you a new pair of shoes. "We like that all our runners wear the same shoes, just for consistency."

"But, this isn't the brand I usually wear", you stammer, "and wait...these are size 12, and I wear a 10".

"Yes, we know. We like everyone to wear the exact same pair of shoes, so that we all are wearing the same pair of shoes."

"That doesn't make sense, though. I have a different foot type than the other runners", you plead. But it's no use. You put on the shoes.

Finally, you make your way over to the starting line, but right before you get there you bump into the club's running strategist. He hands you a shiny, color pamphlet with a bunch of nice info-graphics in it. "Please read this before you run. It explains our micro-burst strategy for racing", he says. "Basically, we'd like you to sprint for 30 seconds, then walk for 30 seconds, then sprint again, then walk, and so on."

"Wait, what?", you question. "Does that work? Do I do that for 26 miles? And have you actually tried this before? Are you even a runner?"

"No...well...I used to run 15 years back, but I don't do that anymore. Running is kind of, you know...low-level. I do read a lot about running though, and now I just help runners by telling them about best practices. Anyway, you better get to the race, it's starting in a minute."

"Ok, whatever." You're beginning to feel a little jaded by all of this encroachment, and you just want to get to doing the thing you enjoy: run.

With just a few seconds before the gun goes off, you see the head coach out of the corner of your eye. He screams out, "we need you to run this race in under 2 hours. Can you commit to that?"

"Are you crazy?", you shout back. "Even without all this crap, the fastest I could possibly run it in is a 3:30".

"I'm sorry, but I need you to commit to running under 2 hours. Can you do it?"

"Are you asking me?"

"Ummm...sure, whatever. I'll run it in 1:59." you shout back.

"Great. I'll mark you down for 1:40, because it shouldn't be that tough. Just push hard. You can do it."

The gun goes off, and right from the start, you feel weighed down by the backpack and you're tripping over your shoes every 5th step. You try a few micro-bursts, but obviously that doesn't work. No fucking surprise. You're getting bitter.

A few racers pass you, and then a few more. Some of the other racers are running solo, but some are racing for other, smaller clubs. Everyone seems so much freer. And faster.

You slog through. By mile 15 the ankle brace has rubbed through the skin. Your shoulders are in knots from the backpack. You look over and see the coach behind a barrier, staring at you and gesturing to his watch impatiently, the international sign for "hurry up".

You push harder, surrounded now only by people from your club. Everyone else has passed you by. You pound through the remaining miles, and get to the finish. The clock says 5 hours and 47 minutes, but it feels like longer.

Despite all the pain, you do feel a strange sense of satisfaction. Sure, you're 2 hours over your usual marathon time, but the fact that you finished it at all is an accomplishment. You high-five the handful of runners from your club who were able to finish, then go get your massage, put your new gear in your backpack, and walk to your car, vowing never to run for a big club again.

...

Afterward

I've done a few stints at large companies, and actually have appreciated and learned from each opportunity - but it was certainly frustrating. For those fighting the good fight at these companies, I have great respect. In my experience, the toughest problems to solve in these environments are often not the technical ones, but the social, political, and organizational ones, and with a little mindset adjustment, I think they can be just as interesting and challenging, but it's obviously hard when what you probably most want to do is just build good software.

In the way of advice (because what kind of blog post would this be without advice?), here are a few survival tips I picked up during my tours of duty in corporate-land. First, be a little bit cunning. Just a little bit though. Not deceitful or malicious, and not to pursue your own ambition, but make the conscious decision that your job is to do what's best for the company, and sometimes this means finding ways around or through the over-bearing policies that get in the way of that. "Forgiveness over permission" seemed to be the mantra of the most effective programmers I met.

Second, use your work to get past the blocks and blockers. If an ivory tower architect tells you that your approach is impossible or sub-optimal, go back to your computer, put in the extra hours, and at your next chance give a demo of how it does work. Or better yet, show your prototype before you get any feedback. In other words, put in the work, and then let it speak for you - try not to get bogged down in the endless meetings and other bureaucratic shenanigans.

Anyway, I'd love to hear any of your tips or tricks for not just surviving but thriving (and dare I say, enjoying) life as a programmer at a big company. Please share!

Surprisingly, however, there's not a lot out there on the intersection of the two - i.e. what's the best way to version REST APIs when doing API-First with OpenAPI? I'm just getting familiar with OpenAPI, but in this post I hope to take a stab at this question.

Do you really need to version?

Probably the best place to start first is to make sure versioning is actually needed for a particular service. Independent of OpenAPI, a good versioning solution takes a non-trivial amount of forethought and development effort, and if you can avoid having to version altogether you can save a lot time and energy. Versioning an API is only necessary when either:

1. You don't own the clients calling your API (e.g. you're providing an API to others)
   
   
2. You do "own" the clients, but for whatever reason (there are too many, etc.) you aren't able to coordinate updates of the service/API with updates of the clients

Basically, if you're building a publicly available API, then versioning is of course necessary. Or even if you have some internally available microservice but it's called by many other clients (microservices, UIs, etc.) from other departments on different release schedules, etc., then versioning might also be crucial since it would be tough to force the clients to update in step with the service.

If what you have, however, is a service that has just a handful of other clients, all of which are owned by your development group and could conceivably all be upgraded/released in step, then it might be better to skirt the versioning issue altogether and just a support a single, canonical but evolving API. In other words, when the API changes, either do it in a way that doesn't break existing clients, or if a breaking change is necessary then go and update all the clients as well to adjust to the API changes, and release them altogether. Sure there are short-term coordination and development costs to doing this, but they may be less than the long-term costs that come with creating and sustaining a versioning strategy and all that comes with that (backward compatibility, deprecation, etc.).

TLDR; versioning is hard, and if you don't have to do it, don't!

When you do need to version

Ok, with all that out of the way, if it is determined that you need versioning, then how do you do it using OpenAPI and API-first? The central concept in API-first is obviously is the API spec itself, from which everything else is generated (clients, server stubs, documentation), so the big question is what to do with this? Specifically, how many API specs should there be (e.g. one per version, etc.)? And what do they encompass? It seems to me like there are two main options: (1) put all versions in one API spec, or (2) create a new API spec per version.

The first option, putting all versions of an API into a single api spec, does seem to have a few advantages. First, everything is in one yml file (i.e. paths for all versions of resources) and from this single spec, everything can be generated (e.g. clients, server stubs, etc.). Clients, services, stakeholders, etc. just need to look to one artifact for what they need, regardless of what versioning they're using.

Additionally, this allows for easily versioning at a resource-level, and not an API-level. For example, say there are just three resources:

/resourceA
/resourceB
/resourceC

...and say only one of the three changes, resourceA. With a single API-spec it's possible to only version the one that changed, resourceA, such that it would have both a /v1 and /v2, but the other resources, resourceB and resourceC, just have a /v1.

There are a couple major downsides though. First, having all the different versions in one spec will eventually get muddled in that even though it's easy to create new paths for each version of a resource, it's likely different versions of the components (data classes) will need to be created too, which are very similar to each other (e.g. Resource1V1, Resource1V2, etc.). A little messy. Additionally, if you're using the spec to generate clients, then clients could wind up with all versions of paths to choose from (unless tagging or whatever is used to generate a client with only v1). Tagging can be helpful here to only generate some portion of the API spec for a given client, but this could be tricky.

The alternative is to create a new API spec for each version, and within this there are two sub-options: version at a resource-level or version at an API-level

The alternative here is enforce that the API is versioned altogether. From the perspective of the organization of the API spec, this seems much cleaner, but there will probably be a lot of redundancy across versions. For example, again if only resourceA changed from v1 to v2, and resourceB and resourceC stayed the same, you'd have to copy-paste over the entire specification for resourceB and resourceC into the v2 spec document even though they didn't change. This duplication just feels wrong, but if you don't like the other options, it might be a necessary evil though.

It's probably important to mention here that there is a version property in the API spec itself, and while it would seem that this should define the version of the API itself, it actually is intended to define the version of the API specification document (a subtle nuance that was a little confusing to me).

There's plenty more to discuss, but I just wanted to broadly paint the different options. The take-away for me is that versioning is not easy, and when you can, it's better to evolve an API either in a way that is non-breaking for clients or to evolve the API and clients at the same time. Creating an entire new version of an API should only as a last resort. I'd love to hear anyone else's thoughts!

At first, the metaphor of technical debt seems so crisp and clear. It's the perfect way to explain to your manager, product owner, or fellow developers the impact of all the copy-and-paste code you see in the code base or those horrible thousand-line functions nested 8 levels deep.

"In order to make that last release, we had to cut a few corners. These cut corners are like 'debt' , and at some point we'll have to pay it back or we'll go bankrupt"

At this level, the concept of technical debt is absolutely effective. Everyone gets it. Just like any individual person, a software development team too needs to manage their "finances". Sure, it may be ok to take out a line of credit to "build that addition" (or whatever), but as a standard way to operate it's not sustainable. Pay now, or pay later.

So what's the problem?

The devil is in the details, as they say, and although the metaphor of "debt" makes a lot of sense at the conceptual level, things get fuzzy real quick when you try to put your finger on exactly what it is. This matters because when it's difficult to define something, it's really tough to measure or manage it, and this is exactly what we want to do with technical debt. Once we accept that we have this "debt" in our systems, we naturally want to know how much we have, whether that's a lot, and how long we have until the bank repossesses our system. The trouble is, there are so many different interpretations of technical debt, this is anything but trivial.

Sure, there is a consensus definition that seems tenable and intuitive at a high-level, but it frays with just a little scrutiny. To see what I mean, let's start with the standard definition of technical debt. Paraphrasing from the Wikipedia:

Technical debt is the cost of additional rework caused by choosing an easy solution now instead of a better approach that would have taken longer.

The first thing to note is that taking on debt is a decision we make. A deadline looms, and so we consciously choose expedience over "doing the right thing". We've all been there, and calling this "debt" is an apt description.

But wait. Is it always a choice? What about those cases where we unknowingly implement the inferior solution? Maybe we were new to the language or framework, and had no idea there was a better way. Or maybe we were just genuinely clueless! Martin Fowler considers this technical debt also, but classifies it as inadvertent debt. Ok, so now tech debt isn't necessarily a conscious choice, but it is still something we do to ourselves...

But maybe not (!), because it's easy to think of cases where it wasn't our doing. For example, what if we were moving along merrily and then some underlying framework we were using changed, and we now need to upgrade or migrate (AngularJS anyone?). Some call this accidental debt - it's something we have to address and it'll get worse the longer we wait, but it definitely wasn't a choice - it happened to us because of age or decay. But again we're further away from the original definition...

And then is tech debt only about maintainability? Could it be related to any "ility"? For example, if I quickly bang out some naive/slow algorithm but add a // TODO fix this to make it faster, is that tech debt? It seems to fit the criteria of expedience over doing what's "right" in the long run, but it's about performance and not maintainability. And if this too is technical debt, why couldn't we consider some sub-optimal solution for security or resilience tech debt as well (e.g. the system doesn't handle failure now, but could with some rework)? And while we're at it, is lack of code comments tech debt? Or lack of documentation in general? And so on and on...

Maybe this seems like pedantic nit-picking...but that's kind of the point! As we ask reasonable questions about what technical debt is and what it is not, we stretch at the seams of the concept in general, and it begins to unravel to the extent that it seems like either everything is tech debt, or nothing is tech debt (a position held by some) and at this point the entire term can seem almost vacuous.

So where does this leave us? I think there are a few options. One is to just admit that the more we allow for new and different edge cases of technical debt (e.g. accidental debt, inadvertent debt, reckless, etc.), the more we water down the entire concept in general until no one really knows what it is, and it begins to lose its effectiveness as a metaphor. Just hold the line on the original definition and we'll be better off.

Another option though is to see it as just a metaphor, and then at a backlog/management level, treat all work as simply stuff that needs to get done to improve the system. Whether a given work item is there because we cut a corner a few months ago or because our version of Java is being sun-setted next month doesn't matter - it's just all work we need to prioritize and then do. Basically, use "technical debt" as an explanatory term, but at a practical level just manage it on your backlog with everything else.

In the end, I'm curious what you think. Am I missing a key distinction? How do you use the concept of technical debt? Is it helpful?

As a developer, you often find yourself in a situation where someone is describing some new system to be built, and you're trying to wrap your head around exactly what "it" is. Maybe it's a client with a potential new project, or maybe just a friend with another one of his "can't-miss" ideas for a new app. Regardless, the goal is essentially the same: through a simple conversation, you want to quickly size things up and help them figure out (a) whether it's technically feasible, and (b) if it is, how it could be built.

Early in my career, I'd usually cut right to the chase (or so I thought). I'd get a basic sense for what the system needed to do, and then I'd start sketching out my ideas for the tech stack:

"Ok...it's a B2C commerce web app here...pretty standard...so probably React, NodeJS, and Mongo."

Or whatever. The point is, as an opening gambit, this wasn't very effective. While it might convey some level of confidence to the "business" owner that I was a capable technologist (I mean, hey, I threw out some good buzz words), it wouldn't actually buy me much in terms of a real understanding of the risks, constraints, and goals that should drive the architecture. If the project did come to fruition, I'd quickly realize that my "design" had some serious holes, and I'd have to scramble to fill them in.

(As an aside, I think this is a common pitfall for developers. Although we like to fashion ourselves as "problem solvers", too often we live up to only the second half - we're just "solvers". The exact "problem" that we're addressing isn't always clear, well formed, or even real, because we were too quick to jump in with our own ideas and assumptions, instead of slowing down to first listen and understand.)

Nowadays, with a little more experience, I try (at least in these early conversations) to only ask questions, and hold off on proposing anything too early. If architecture is about the hard things, my goal is to hunt down what these things are. Only when I've mapped out the terrain do I try to map out a route forward.

For whatever it's worth, here are a few questions I start with:

1. How many users (or consumers) are there, how much will they use it, and are there patterns (or spikes) to their usage?
2. What devices would access the application with (e.g. mobile, desktop, voice, etc.)?
3. What is the sensitivity or importance of the functionality or data that this system provides or exposes? For consumers, is authentication alone sufficient, or does access to certain pieces of the application need to be controlled or managed (i.e. access control)?
4. Is any data persisted? If so, what type of data is this (structured, unstructured, etc.)? How much is there? How does the data get into the system (e.g. user entry, telemetry, etc.)? Is data created, read, updated, or deleted? How important is integrity or consistency?
5. What other systems does this system need to interact or integrate with? What is the nature of the communication (e.g. real-time, batch feeds, etc.)? What type of security is necessary?
6. On what hardware does the system run? Is it a kiosk, embedded system, server-based, etc.? Can it be hosted on a public cloud, or does it need to be managed internally? Who owns and manages the machines on which the system runs? How will it be deployed? How will updates be delivered?
7. How critical is the application? Can any downtime be tolerated, and if so, when? What is the cost of downtime?
8. Who will maintain the application when it's in production? What tools will they have available to monitor or verify the health of the system? Will they be able to
9. What is the timeline for which the system needs to be built (or new features need to be delivered)? What is the development capacity?

In the end, this is software, and so there are innumerable facets, variables, and complexities to explore (and that could trip us up), and so there are plenty of other questions that might need to be asked. These are just some starting points, but where the conversation flows from there depends on the system.

The root of the problem, I believe, is what Daniel Kahnaman calls What You See Is All There Is., which explains, in a nutshell, that we're hard-wired to want to make decisions quickly, using only the evidence we have at hand and some basic heuristics. Slowing our thinking down would take time and discipline, and so instead we try to substitute complex problems we don't fully understand for simple ones we do.

In the case of reuse, our intuition is simple and strong, represented by that hard-to-shake physical world analogy of software as a "wheel" not to reinvent. And it's this comfortable mental model that we often fall back on when making decisions about reuse. The problem is that this basic intuition about reuse is flawed, or at least woefully incomplete. Let's see how...

(A quick caveat: my argument here is around large-scale reuse, not at the method or function level. I'm completely down with DRY at the lower level of granularity. Also I'm thinking of reuse as the leveraging of some service/library/etc. that was built internally vs. acquired externally. I'm not recommending building your own JS MVC framework! Ok, now back to the originally scheduled post...)

The Intuition

Imagine there is a system, A, that contains some logic, C, within it. Soon, a new system, B, is to be built, and this system too needs the same basic logic, C.

Now it stands to reason that if we just split C out from A, it could then be used by B without the need to re-implement. The savings is therefore equal to the cost of one instance of C - i.e. it only had to be built once, and not again for B.

Further, as more systems are identified to have this same common code, the benefits of this extraction and reuse will be extended to those systems as well in a linear fashion, such that for every new system that reuses C instead of re-implements it, there is another savings equal to the implementation cost of C.

Again, the logic here seems simple and seemingly unassailable - why would any organization choose to build multiple instances of C rather than just build it once and then reap the benefits of reuse. The problem is that there's more to this picture, and what can look like slam-dunk ROI up front can turn into an costly straight-jacket down the road. Here are a few ways in which the basic intution of reuse breaks down...

The Reality

The first problem is that of extraction. Our intuition is that C should snap apart from A like a piece of lego - nice and easy. The reality, however, is that disentangling common code can be a little like pulling a piece of spaghetti from your bowl of pasta, only to find that the entire dish is just one long noodle. Of course it's not usually that bad, but in code, there are lots of hidden dependencies and connections, and the initial conception as to the scope of C grows as you begin to unwind it. The effort is almost never as easy as you expect.

Moreover, almost always, C needs other things to do its job (e.g. other libraries, utility functions, etc.). In some cases these are shared dependencies (i.e. both A and C need them) and some cases not. Either way, the simple picture of A, B, and C can begin to look less so. For the sake of this example, let's assume both A, B, and C each use a common library, L.

Another problem is that of variation: different consumers of C will often have slightly different requirements for what it should do. For example, there may be some function in C that needs to behave slightly differently if A called it than if B did. A common solution for these cases is parameterization: the given function takes some parameter which lets it know how to behave given who called it. This can work, but it increases complexity of C, and the logic gets messy as well, as the code gets riddled with if blocks like "if called from A then do this block of logic".

Now even if C is a case of perfect reuse for A and B, this type of variation almost always is necessary as new systems, say D and E, come along. They may want to use some of C the way it is, but then need other parts to change in subtle or not-so-subtle ways. Again, each new accommodation that needs to be made within C represents extra complexity, and so what used to be something fairly easy to grok from the perspective of a developer using it, now becomes a lot trickier as C morphs into something that must satisfy the needs of D, E, F, and so on. Which leads to the next problem...

As this complexity increases, developers have a tougher time groking what C does, and how to use it. For example, a developer of A might not understand some parameter for a function of C, since it's only relevant for systems E and F. In most cases, some level of API documentation is necessary (maybe Swagger, Javadoc, or more) to explain the inputs, outputs, exceptional conditions, and other SLAs/expectations. And while documentation in general is a good thing, it's not without its problems (e.g. keeping it up to date, etc.).

Another implication of increased complexity is that it can be harder to maintain quality. C now serves many masters, and so there are a lot of edge cases to test. Further, since C is now used by many other systems, the impact of any given bug is amplified, as it can pop up in any or all of the consuming systems. Often, for any change to C, it's not enough to test just that shared component/service, but some level of regression testing should be done as well for A, B, D, and all the other systems that depend on it (whether the change to C is actually used by that system or not!).

Again, since we're talking about reuse at a non-trivial scale, it's probably the case that C will need to be developed by a separate team of developers, which can lead to a loss of autonomy. Separate teams usually have their own release schedules, and sometimes their own development processes. The obvious implication is that if A's team needs some enhancement in C, it probably needs to work it through C's process - i.e. a champion of A needs to provide requirements, advocate for its priority, and help with the testing. In other words, team A is no longer in control of its own destiny with respect to the functionality that C implements - it is dependent on the team that delivers C.

Lastly, when C is upgraded, by definition there are now different versions. Depending on the nature of the reuse, this can present different issues. In the case of build-time reuse (e.g. library), the different systems (A, B, etc.) could stick with their working version, and choose to upgrade when optimal. The downside here is that there are different versions of C in the wild, and it's not unlikely that one bug might lurk in each of the versions, and need to be patched in each. In the case of run-time reuse (e.g. microservice), C would either have to support multiple versions of its API in one instance or it could just upgrade without consideration of backward compatibility and thus force A and B to upgrade along with it. In either case, the robustness and rigorousness of the processes and organizations to support this reuse are significantly increased.

Conclusion

In the end, my point is not that large-scale reuse should be avoided, but rather that it's not as easy as our intuition suggests. Reuse is hard, and while it may still deliver benefits that outweigh its costs, those costs need to be realistically considered and discussed up front.

Even after careful consideration, if large scale reuse is the right course of action, there's still a decision as to the nature of reuse. My experience tells me to be careful of the dependency arrows. Reuse where the "reuser" is in control is almost always easier to implement and manage than reuse where the reusable asset calls the system. In the example above, having C be a library or microservice puts A and B in the driver seat, and this, in my opinion makes for quicker implementation and less management/coordination in the long run.

Flipping the dependency arrows, and making C a framework or platform, changes the onus of control. Now A and B are beholden to C. This type of reuse is not only more difficult to build (and get right), but results in greater lock-in down the road (i.e. A and B are totally dependent on C). A great adage is that "a library is a tool, a framework is a way of life".

In the end, I'd love to hear your feedback or experience with large-scale reuse. When does it work out and when does it fail?

It's the beginning of the sprint. Everything's been planned, estimated (if that's how you roll), and assigned. You have a few features on your plate, and are eager to get going. Starting with the most critical one, you dig in for a day, get a plan together for how to tackle it, and begin coding.

Another day passes and you get about halfway through when the project manager swings by your desk. "Hey", he says, "I know you've already started, but a business priority has changed, and we need you to work on this different feature instead. Finish this new one off now, and then you can get back to your original work".

I'm not sure about you, but this drives me freaking crazy (at least it does when it happens often). But should it? I mean, I'm being paid by this company, and it's up to them to set their own priorities, right? If they decide there's more value in a different feature, that's their call. Further, I'll get back to the original feature eventually, and if I don't, who cares - I learned a few things in the process of digging in, and probably have a better grasp of the application as a result. What reason do I have do be frustrated? And yet I am...

This internal tension has plagued (or at least puzzled) me forever. Why do I hate leaving tasks unresolved so much? And I think its not just me - most of my colleagues seem to get just as frustrated when it happens to them. But why? Recently, while reading the book Irresistable, I found an explanation.

In psychology, there's a well known cognitive bias called the Zeigarnik effect. It states, simply, that an incomplete or interrupted task will be remembered better than a complete one. The phenomenon was coined by Bluma Zeigarnik, a psychologist who noticed that waiters had better recollections of orders that were unpaid. As soon as an order was completed, the waiter effectively flushed it from memory. In her paper Finished and Unfinished Tasks, she explains:

"When the subject sets out to perform the operations required by one of these tasks, there develops within him a quasi-need for completion of that task. It is like the occurrence of a tension system which tends towards resolution. Completing the task means resolving the tension system, or discharging the quasi-need. If a task is not completed, a state of tension remains and the quasi-need is unstilled."

This hits the nail on the head for me. When I start development of a new feature, there's a non-trivially strong force within me that wants to drive it to completion. If I'm asked to table that feature in favor of something different, my brain can't completely let go of the original task and so I can't fully focus on the new task at hand. The details of the original task are still churning in some background process in my brain.

Further, my guess is that the Zeignarnik effect is probably cumulative as well. The more tasks interrupted, the greater the tension and frustration. And for some, this is the rule rather than an exception. You get into work intending on finishing off some feature, but then get pulled into a meeting to discuss something different, and then get pulled out of that meeting to work on a production defect, and so on until the day ends and almost nothing was fully resolved. Ultimately frustrating.

Adjustments can (and probably should) be made in an organization to avoid this effect, since the effectiveness, mental clarity, and possibly even sanity of their developers depend on it. First, of course, we should strive to limit interruptions and let developers close open loops before forcing new ones upon them. Many teams have already started doing this through the adoption of Kanban and work-in-progress limits. Additionally though, in environments where interruptions are rampant, perhaps tasks could be designed to be as fine grained as possible, allowing developers to finish something before being pulled off to work on the next. Lastly, it's possible the best solution might just be to strive to overcome our innate mental wiring - when you get frustrated over switching away from a task, just acknowledge it's just the Zeignarnik effect, and do what you need to do.

Anyway, I'd be interested to hear about your experiences. Is the Zeigarnik effect real, and does it cause frustration? What do you or your team do to address it?

As an example, maybe your central application is some transactional web app, and there's some data cleanup you need to do at the end of the day - some user transactions, say, that are helpful to keep for a day, but can be purged at midnight. Assuming this is a reasonable candidate for a batch job (bear with me), the question then is, what is this batch job? What I mean is, is it external to the transactional system from which the data was created, or is it embedded within it?

It's probably obvious that, from a scalability and performance perspective, the best solution is the external one, since any resources (CPU, memory, etc.) consumed by the batch job are not subtracted from the core transactional system. (though, yes, they both share the DB...so maybe not the best example!)

But is there ever a case where it makes sense to run batch jobs within a transactional system? Though an architectural purist would probably say "no", there can be a few advantages to this approach:

1. Simplicity: Depending on your organization/environment, procuring a machine and resources to run batch jobs can be a challenge, as can be the management of the scheduling and execution of these processes. Sometimes the easiest thing to do is just include the batch processes in the thing (i.e. deployable unit) you have. For big organizations (where bureaucracy and overhead are high), this is probably the path of least resistance.


2. Code Reuse: Often times, the batch job needs to do some of the same types of "stuff" the transactional system does. For example, it may query the same tables, shove them into the same domain model classes, and execute the same business logic. To avoid code duplication, these code assets would need to be externalized to some common library for both the batch job and transactional system to leverage, and the work of extracting code into new libraries is non-trivial.

If you do decide that in your case, these advantages of running batch jobs within a transactional system outweight the drawbacks, then it's important to note a crucial snag: clustering. When the batch job is embedded in the transactional system, and there are multiple instances of the transactional system deployed in a cluster (for either scalability or reliability/availability), then some mechanism needs to be in place to ensure that the batch job gets kicked off on one and only one node in the cluster.

There are a few possible solutions to this problem, some more reliable/elegant than others. One naive option is to just configure one node in the cluster to be the one that runs the batch job (e.g. "if hostname='server1.acme.com' then run batch job"), but this has the obvious flaw that if this server is down, another node in the cluster wouldn't know that it needs to pick up the slack. JBoss has a more robust form of this solution with its HASingletonDeployer.

Another workable option is for some external scheduler to kick off the batch job, with a request (either via MQ or even HTTP) to the transactional system. This option is workable, but has some downsides - the transactional system needs to expose endpoints (or consumers) to listen for requests to run the batch job and the remote scheduler needs to be sophisticated enough to know that if the job wasn't picked up (or run successfully) on one node, the request needs to be sent to the other.

Probably the best solution for this problem I've found, for the Java side at least, is a library called ShedLock. It's laser-focus-designed specifically to solve this exact problem. All nodes in a cluster would be able to run the batch job, but they would use a central database to obtain a lock before running it, where only the node that has the lock runs it, and the rest skip it. The configuration is super easy, and rather than regurgitate already great documentation, just check out their github.

Anyway, this was a problem that we recently confronted, and since I didn't find a ton of good content out there I figured I'd write this out. Hope it helps someone!

From the Wikipedia, a code review is defined as "a systematic examination of computer source code intended to find and fix mistakes overlooked in the initial development phase, improving both the overall quality of software and the developers' skills". A great deal of evidence, both empirical and anecdotal, has supported the effectiveness of code reviews, and most would consider them to be a best practice in software development. In general, the main benefits are:

• Improve internal quality of the system (e.g. maintainability, extensibility, etc.)
• Improve external quality of the system (e.g. reduce bugs, improve quality)
  
• Facilitate the transfer of critical project-related knowledge amongst team members.
  
• Provides a format for the mentoring of junior developers.
  

Some less obvious or widely touted, but equally valuable benefits from code reviews are:

• Spot opportunities for re-use, thereby reducing code redundancy.
  
• Enforce consistency in development practices and application of coding styles and patterns.
  
• Provide managers with a measure of progress during the construction phase.
  
• Keep developers motivated to perform higher quality and quantity work via a peer pressure mechanism.
  

Fagan Inspections are typically viewed as the canonical form, however code reviews can come in a number of flavors:

• Fagan Inspection: The most formal and rigorous of code review techniques. Involves the public review of one developer’s source code by a set of reviewers, facilitated by a moderator.
  
• Walk-through: A less formal, perhaps impromptu meeting where developers read through code in real time with the intent of detecting errors.
  
• Pair Programming: A practice whereby two developers work together at one keyboard. Note I'm not a huge fan of pair programming.
  
• Pass-around: Code is distributed and reviewed independently by developers, and comments are then provided to the author.
  
• Automated Review: Code is mechanically inspected by static source code analysis tools.
  

Recognizing this, that the “fit” of the code review process in an organization is critical to its success, it would be unwise to rigidly recommend one process for developers to dogmatically apply across all clients. Instead, what is needed is a code review process that is effective “out-of-the-box” for our typical projects and customers, but ultimately adaptable, acknowledging that no project or customer is ever really “typical”. Additionally, the process must also be relatively familiar to most developers (e.g. similar to Fagan, etc.), must not be overly demanding or time intensive, and must mesh well with other SIP processes (e.g. RUP, Scrum, etc.). The process defined below aims to achieve each of these goals.

Process

The diagram below illustrates the baseline process for a code review. Note that for the purposes of this process, there are just two roles – authors and reviewers. Typically, the set of reviewers will consist only of developers on the project team, but could also include other project stakeholders (e.g. for validating business logic, etc.).

1. Identify Files: After being informed of the segment of code to be reviewed (see "What code to review"), the author identifies a set of source files (e.g. configuration files, UI code, scripts, etc.) and possibly requirement or design documents (if they are germane to the review) and emails this list to all reviewers along with any caveats (e.g. "don't look at method X, because it's not finished", etc.). This list of files should take no more than one hour to compile and should be sent at least 4 days prior to the meeting, giving reviewers enough time to inspect the code.

2. Review Code: Upon receiving the list of files from the Author, each reviewer reviews the code on his own. Among other things, reviewers are looking for functional correctness, style, clarity, extensibility, reusability, and performance. For each project, a code review checklist can be created (in the SDG or on the wiki) and then referenced by the reviewer to ensure that code adheres to the project-specific conventions and standards. The author records all issues, suggestions for improvement, or kudos found in the code, organizes them by file, and then emails this feedback to the author at least one day prior to the meeting. Reviewers should cap the amount of time spent reviewing code at one hour (data suggests that there are diminishing returns after 60 minutes, and investing more than an hour is too taxing on the individual Reviewer – he has his own work to do too!).

3. Review Feedback: After receiving all feedback from Reviewers, the Author should examine each comment and determine (a) whether the comment should be addressed (e.g. fixed, refactored, etc.) and (b) whether the comment merits extra discussion among the group. For example, feedback about a missing method comment is easily addressable (just add the comment!) and doesn’t warrant a full discussion. On the other hand, a question about a coding convention may indicate the lack of a standard, and should be discussed to garner consensus. The author should then compile a list of all comments from the Reviewers, mark those comments that need to be discussed, and print or email a copy for each Reviewer prior to the meeting.

4. Discuss Code: Only when the Reviewers have examined the code and the Author has filtered the list of items to discuss should everyone get together and talk. At the meeting, the Author facilitates the meeting, stepping through each discussion item and making notes of any resolutions or decisions. The meeting should move briskly, but should not be rushed – good, substantive discussion is the goal, and this takes time. As a rule of thumb, the meeting should last no longer than one hour. Any items not discussed or resolved should be tabled for another meeting or figured out independently by the Author or tech lead. Meetings that last over an hour typically leave participants feeling frustrated and less interested in doing a code review again.

5. Implement Changes: Within a few days of the meeting, the Author cycles through the list of comments from the review and makes changes to the code where appropriate. In some cases it is useful to post the feedback, resolutions, or decisions to a wiki, but only if there is time (this is an agile process!).

Best Practices

The following practices have been proven to be generally helpful when performing code reviews:

What code to review: In general, the project’s technical lead is the most appropriate person to select the code to review, however the project manager or team as a collective could decide as well. When selecting code to review, the following general rules should be considered:

• The first review of any project should target a senior developer’s code, setting the tone that no developer writes perfect code and any developer can find valid issues in any other developer’s code.
• Critical or risky pieces of code (e.g. key business logic, integration points, reusable libraries) should take precedence over non-critical, safer pieces of code.
• Over the course of a project, each developer should have their code be reviewed. No one on the team should be absolved, neither the most junior nor the most senior.
• A segment of code to be reviewed should include no more than 10 files.

Use static code analysis tools prior to review: Static code analysis tools like PMD and Checkstyle can be extremely useful in rooting out syntactic and stylistic issues in the code more quickly than a manual review. See automated review for standard configurations and best practices.

Be careful of feelings: Developers by their nature often feel quite attached to their work. Criticisms of code can easily be perceived as criticisms of the person, and so it’s common that authors leave feeling hurt, angry, or scorned. Because of this, reviewers need to be keenly aware of the social component in code reviews, being careful to praise more than criticize, focus on the code and not the person, ask questions rather than make statements, and in general be tactful and build trust. As Karl Wiegers says, “bugs are the bad guy in the review, not the author”.

Document coding standard: Oftentimes code reviews will expose differences in opinion about coding standards, best practices, or patterns. These disagreements are extremely useful, as they are the first step toward consensus. In fact, code reviews are sometimes the only opportunities developers get to discuss and resolve such issues in the course of a development cycle – and catching them quickly is helpful for ensuring coding consistency and reducing redundancy. As agreement is reached, however, it is important to document this in a knowledge repository (e.g. Wiki) or in an SDG, SAD, or SDP to ensure that the conflict can be quickly resolved in the future.

Find a management advocate: In staff augmentation projects, garnering support from management is imperative - without it, the time spent cannot be justified (e.g. on time-sheets, etc.), developers will resist or thwart their adoption, and the practice will degenerate as other time pressures come to the fore. Oftentimes, managers will be supportive of a code review practice, however, when they are not, it is helpful to present the benefits (both intangible and hard data) and note that code reviews are a common, proven practice adopted by countless other mature software organizations. If support is not there, code reviews can still be on off hours (e.g. lunches, early mornings, etc.).

Assign a watch-dog: Most developers know that code reviews are good for them, but like exercise or vegetables, they're easily neglected. Therefore, it's important to have one person responsible for managing, coaxing, and otherwise nagging developers about code reviews. This most often is the responsibility of the Technical Lead on the project, but could also be the project manager.

Review code every week: A common practice is to review code prior to the completion of a requirement, using it as a code quality gateway that developers must cross through before a final check-in. While this seems ideal in theory, in practice the logistics often become intractibly complicated. In most development organizations, developers complete features at the same time (i.e. before the end of the iteration), and so each developer needs each other developer to review code at the very time that no one has any time to spare. First one review gets postponed, then another, and before you know it, there are no more reviews. A more sustainable practice is the rotating code review, where each week a different developer's code gets reviewed by the team. While this doesn't guarantee that code is reviewed at the most optimal time, it does set a nice cadence and predictability to reviews, and helps install a healthy amount of fear in developers which in turn foments better coding practices (i.e. "it's possible that my code will be reviewed tomorrow, so I better write good code today"). In general, the more often you do code reviews, the easier, quicker, and more effective they will be.

Account for Code Reviews in Estimates: Using the process defined above, code reviews require only a minimal time investment from each team member - about 5 hours for the author and 2 hours for each reviewer. In most cases, for both project work and staff augmentation, this time can be factored into the normal project schedule (like unit testing, documentation, etc.).

Hold review meetings over lunch: A good way to ensure that code reviews don't intrude on normal, burn-downable project work is to hold the meetings over lunch. This also helps make code reviews less formal and a bit more fun.

Use code reviews as a vehicle to discuss architecture and design: On many projects, there is no separate, allocated time for team members to discuss design or architecture issues. If this is the case, Code Reviews can serve this purpose as well. As a discussion about one particular method unravels into a more abstract discussion about performance or extensibility, it's useful to not cut this discussion short, as it can expose important gaps in the architecture or design.

Code reviews should not be performance reviews: In the rare cases where project managers, business analysts, or even clients are involved in a code review, it's crucial for the tech lead to ensure that all participants understand that the code review should never be used as a way to assess the author's competence or performance - it should only be used as a tool for improving code and sharing knowledge. A good code review depends on open, honest dialog, and this is subverted when participants feel as though they are being critiqued.

Resources

• Code Complete, Steve McConnel. Chapter 21.3: Formal Inspections
• Peer Reviews in Software: A Practical Guide, Karl Wiegers (November 2002)
• Part of Your Complete Breakfast: Code Review is a Source of Essential Vitamins and Minerals, Stephanie Lussier (2002)
• Pair Programming vs. Code Reviews, Jeff Attwood (November 2007)
• Code Reviews without the Pain, Robert Boque
• Seven Truths about Peer Reviews, Karl Wiegers
• Code Reviews, Srivaths Sankaran
• What Makes a Code Review Trustworthy?, Stacy Nelson and Johan Shumann

Evidence

In Code Complete, Steve McConnell describes specific studies that indicate that code reviews do indeed improve the overall quality a software system. Below are some of the more compelling statistics in favor of code reviews:

• “Individual inspections typically catch about 60 percent of defects, which is higher than other techniques except prototyping and high-volume beta testing.”
• “The combination of design and code inspections usually removes 70-85 percent or more of the defects in a product.”
• “On a project that uses inspections for design and code, the inspections will take up about 10-15 percent of project budget and will typically reduce overall project cost.”

In addition, Karl Wiegers cites some compelling statistics from industry, in Seven Truths about Peer Reviews:

• Hewlett-Packard’s inspection program measured a return on investment of 10 to 1, saving an estimated $21.4 million per year. Design inspections reduced time to market by 1.8 months on one project [5].
• Inspections contributed to a ten-fold improvement in quality and a 14 percent increase in productivity at AT&T Bell Laboratories [7].

Who knows what the future holds, but at least at this moment, it's safe to say that Amazon is killing it with Alexa. Just 6 months ago, the total number of Echos, Dots, and Taps sold were around 5M and the number of skills available about 5K. Today, both of those numbers have almost doubled, to 8.2 million devices and 10k skills, respectively. What's more, it seems like every day now there's some new partnership, integration, or spin-off product to boast of - like the unveiling of the Show (voice + screen), the integration with Ikea, or the release of the first built-in Alexa phone. Even the "just-plain-fun" stuff like the Seattle Mariner's skill for their luxury boxes or the Silver make it clear that Alexa has made it into the zeitgeist.

So with everyone jumping on the Alexa bandwagon, the question for software developers is, should we jump on as well? In other words, is it worth it for us to invest in acquiring the knowledge and tools to build Alexa applications (or skills)? And if we do, what would we build? What are the "right" applications for this platform?

To answer these questions, it's probably helpful to first step back and take a look at what Alexa as a platform has going both for and against it. In my opinion (for whatever that's worth), the matter of whether Alexa will rule the (voice) world is still open - there are reasons for both optimism and concern. Here are my 2 cents...

What Alexa has going in its favor

As already covered, there are a ton of fun things happening in the world of Alexa recently. It feels a bit like the early days of the internet or of IOS apps, where people are just throwing things at the wall to see what sticks. There's a sense that it could be big, but it's not yet clear how or in what way. And this uncertainty and openness is what's so exciting. As developers, the opportunity is there to be a "first mover" - to make a huge impact with a relatively simple idea (if we can just come up with the right one!).

Even better, the barriers to entry for building are pretty low. Having been on the market for almost 3 years now, the platform is stable and rich, there's plenty of great documentation, a vibrant community, and ample choice from a tech stack perspective (with frameworks for Java, Javascript, and others). Moreover, from a conceptual perspective, there's not all that much there. Compared to what what you need to learn to build an IOS or web app, building a voice based application is a snap. Sure, there's some new jargon (like "utterance", "intent", "slot", etc.), but with a few hours of study you're basically there.

What Alexa has going against it

Given all this (millions of users, plenty of room in the market, and a low cost to build), why wouldn't we all jump in? Well, there are a few reasons to be wary, I think.

The first cautionary flag is the "stickiness" of Alexa. It's true that everyone on your block just bought an Echo, but that doesn't mean they're actually using it. A report from VoiceLab recently showed that whereas for an Android or IOS app there's about a 12 percent chance that a user will still be using it in 2 weeks, the percentage chance for an Alexa skill is only at about 3 percent. Further, of the 7,000+ skills available (at the time of the report), 69% have only zero or one review. These stats seem consistent with what I hear anecdotally from friends and family. The novelty of Alexa can wear off pretty quickly, and within weeks she can go relatively unused or might even end up relegated to the desk drawer.

If this is the case then, and there is indeed a retention problem, it begs the question: what's the reason? Is it too few useful skills or something inherent to platform itself?

If it's about the quality of skills, then reason is probably obvious. As of yet, there isn't a clear way for developers to monetize their creations - everything on Alexa is free. As a consumer, this is great, but as a developer not so much. Am I really going to sink dozens of hours into building a skill without a hope that there could be some (even meager) pot of gold at the end of my development hours? I'm not so sure. And this seems clear from even a quick perusal of the skills available. There are a ton of banal "fact of the day" skills, but very few skills of any depth, quality, or creativity (though there are some). Basically, it seems that developers are intrigued enough with the technology to dabble a bit, but without a proper incentive, very few have taken the time to build anything of real value.

It could also be the case, however, that the platform itself is to blame. Many have voiced their concern about some of the inherent limitations of Alexa. Among the major gripes are: an inability to push notifications to the user (pretty much a necessity for social apps), no coordination between multiple Alexa devices (e.g. keep my music in sync on all devices as I walk through my house), inability to identify voice, and no raw recording.

Lastly, not all users are impressed with the overall experience. Relative to Siri or Google Assistant, the conversations can seem more robotic and less natural. Users have to phrase things the way Alexa expects, not necessarily how they would typically speak. For example, whereas I might naturally want to say "Alexa, order me a pizza from Pizza Hut", Alexa wants me to say "Alexa, tell Pizza Hut to order me a pizza". Maybe not a big deal in the long run (as we adjust), but in my experience I do find myself having to think hard for the right words to speak - "hmmm, what the hell is that skill I enabled to find my phone again? Is it 'Phone Finder'...or 'Find Phone'...or 'Find My Phone'? Fuck it. I'll just look under the couch cushions again."

What to Build? What's the "Killer App"?

I'm certainly not a Gartner analyst or an oracle of consumer behavior, so to me it seems that the future is anything but clear. Yes, the potential of the Alexa platform seems huge, but there are definitely some glaring red flags as well. Assuming, however, that you're undaunted by these challenges and want to push forward and test your hand at publishing an Alexa skill, the question (of course) is, what should you build? What is the best use of this technology? Is there a "killer app"? And if so, what is it?

Given the ubiquity of smart devices in general, the crux to me seems to be in thinking about the personal trade-off we all make as users in our every day lives. When are we willing (or compelled) to give up the power and richness of our smart phone or desktop GUIs in favor of the austerity and simplicity of voice? Amazon is making the bet that voice is always simpler, but I'm not so sure. When I'm lounging on the couch, for example, would I rather ask Alexa to play that new album (that I can't exactly remember the name of), or just pick up my phone that's sitting 1 foot away from me on the coffee table (if not in my hand!) and make a few taps? In 2017, as humans have essentially reached a singularity with their smart phones, tapping seems almost easier and less conscious than speech (as scary as that sounds).

My sense is that we need to think about situations in which using our screen-based smart devices is just not possible. This seems to be the premise of a great skill by AllRecipes - i.e. when you're in the kitchen, you're hands are dirty, and so it's better to speak than tap. Another great hands-free zone is of course the car, and it appears that Amazon is working on it.

There are also huge (and obvious) wins for the visually impaired, and companies can use Alexa to create an omni-channel experience to tap into this, albeit smaller, market. For someone with low vision, the Alexa skill might be the best (or only) option to order an Uber, even if the majority of people would still prefer to do it via their smart phone.

Lastly, I wonder if the consumer-facing applications of Alexa are being overshadowed by the potential for business applications. There have long been companies that have offered voice-driven functionality to employees (e.g. factory workers, warehouse pickers, health care workers, etc.), and I would think that Alexa's open platform could open up this market more.

In the end, my projections and pontifications might be obvious or off-base, but I do find the Alexa platform to be exciting. I'd like to hear your thoughts though. As a developer, what do you think about the Alexa platform? Is it worth your while to jump in and learn about building applications with Alexa? Or have you already? What do you think will be the most useful skills?

I'm a programmer, a few months shy of his 40th birthday. It's Saturday morning, my kids are home with my wonderful wife (who is pulling my weight on the domestic front), and I'm at a tech conference. It's a session on React Native, and the presenter is convincing us why it's truly the "next big thing" for mobile development. To me, it seems a bit like JSPs of 15 years ago, with all the logic in the presentation code, but I'm "old", so I assume I just don't "get it".

The presenter blows through some slides, dazzles us with some impressive live coding, and then makes his way to the "name dropping" portion of the presentation, where he rattles off about a half a dozen supporting tools that I never knew existed, including something called Pepperoni (seriously). As someone who just recently got the hang of Angular, it all makes me feel a little disheartened. "Here we go again", I think.

Of course I'm not really surprised. Over the past 20 years, I've taken seats on a number of band wagons, and have generally enjoyed the rides. The buzz that comes with a new "disruption" in programming can be exciting - feeling apart of a community of technical innovaters, championing something that will make things a little easier, quicker, cleaner, better. It can be fun. But on this particular morning, at the cusp of 40, I have to admit I feel a little drained. I know this is part of the job - if I want to stay relevant (and well paid), I know that every so often I need to cast out some of the knowledge that I've so dutifully absorbed, and gear up up for the next journey. It's just how it is.

As I think about it though, this regular ritual of my programming career doesn't seem to be a way of life for other professionals. The doctor at 40 doesn't seem to be worried about discovering that all his knowledge of the vascular system is about to evaporate in favor of some new organizing theory. The same goes for the lawyer, the plumber, the accountant, or the english teacher. While there are certainly unappealing aspects to these professions, it's safe to say that for each of them, mid-way through their career, the knowledge they've accumulated is relatively stable, and has afforded them with some increased measure of respect and compensation. In programming though, 20 years of experience does not seem to confer those same advantages.

Two Main Forces

Of course not all is so dismal in our profession - there are so many things to love about being a programmer - but in terms of the never-ending struggle to "keep up", it is an interesting feature that seems more or less unique to our field. Am I right though? Is programming really different in this regard? And if it is, then why? And what does it mean for our career trajectory? I'd like to try to answer all of this (because, why not) in terms of two concepts.

The first is knowledge decay. Everything we know, not just about programming, has an expiration; a point at which it is no longer useful. I learned how to drive a car when I was 16, and for that most part, that knowledge still serves me well. This piece of knowledge could be said to have a long half-life. For many professionals, their domain knowledge also has a relatively long half-life. Sure, new discoveries in medicine may displace some existing procedures, but likely there will not be a major overhaul in our understanding of our biology. When the expiration is long like this, knowledge can effectively be considered cumulative. The doctor is more knowledgeable than he was last year, because everything he learned in the past 12 months built on all that he knew before.

In programming, for good or bad, I'd assert that this is not exactly the case. Putting a (rather arbitrary) stake in the ground, I'd say that:

Half of what a programmer knows will be useless in 10 years.

This could be way off (and there are many caveats of course - read on!)...but it seems about right for me. If I learned nothing else from this point forward, I bet that only about a half of my knowledge could I still use in 2026 (long live SQL!), and the other half would probably be of no use (React Native, perhaps?). Now of course I will be gaining new knowledge to replace the dead stuff, but will it be enough? Will I know more (useful) knowledge in 2026 than I do now?

This brings me to the second concept, knowledge accumulation rate - the pace at which we add new things to our knowledge corpus. In every field, there is a certain threshold of knowledge that must be met in order to be "certified" (or at least hireable), and the early portion of a career is typically dedicated to acquiring this knowledge. In programming, however, because of the fast decay of knowledge, it seems like we never really transcend the "student" period. We know we must always be learning, and this makes the stages of our career a bit atypical.

The Three Stages

If I were to graph an average programmer's knowledge over the course of their career, keeping in mind knowledge decay and accumulation rate, I think it might look like something this:

In the beginning of our careers, in what we could call the eager apprentice stage, accumulating knowledge is relatively easy. Everything is new, and so each experience is a vehicle to gain more knowledge. Moreover, since we're younger, we often have fewer hard obligations, and so we probably don't mind spending a few nights and weekends picking up new languages and frameworks. Lastly, and importantly, the expectations on us from our employers is lower. Everyone understands that we're junior, and so more easily than our colleagues, we can carve out a little time during the work day to fill in holes in our knowledge. This is a fun stage, but there's this persistent feeling that there's so much we don't know.

At some point though we cease to be novices, and we establish ourselves as productive, self-sufficient developers. For the first time, the gap between us and our colleagues (even the ones 10 years our senior!) does not seem so large. This fills us with vim and vigor, and so this is the rising star stage. The investment we made in learning has paid off, and just about everything we know is still useful - i.e. none of our knowledge has noticeably decayed. With this reservoir full of relevant knowledge, we begin to earn the respect of clients, peers, and managers, and with this respect comes titles, salary, and opportunities. Though we don't necessarily see it at the time, this is also an important point of inflection.

It's at this point that two things happen. First, that promotion to "senior" comes with something more than just money: greater expectations. Employers need their star programmers to be leaders - to help junior developers, review code, perform interviews, attend more meetings, and in many cases to help maintain the complex legacy software they helped build. All of this is eminently reasonable, but it comes, subtly, at the expense of our knowledge accumulation rate. The time we used to have to read tech blogs: gone. Second, it's also at this point that we first experience (or at least recognize) a little knowledge decay. Some of what we learned early in our career is now out-dated. All that time "we" (read: I) spent learning GWT? Lost! Essentially, both forces, knowledge decay and knowledge accumulation rate, begin to work against us.

It's at this point where we enter the third and final stage, the ebb-and-flow of the steady veteran. We are knowledgeable and productive, yes, but we also understand that we may actually know fewer (useful) things than we did at a prior point in our career. A non-trivial amount of our knowledge has decayed, and we may not have had the time to accumulate enough new knowledge to compensate. This can be frustrating, and I think it's why it's at this point that so many of us bail for other pastures - management, sales, testing, or (my dream) farming. We realize that it'll require real effort to just maintain our level proficiency - and without that effort, we could be worse at our jobs in 5 years than we are today. There is no coasting.

Humble Advice

This is where I'm at. I still love to learn, but I appreciate that without some herculean effort, I will probably always remain in an equilibrium state hovering around the lower boundary of "expert". I'm ok with this, because I enjoy my personal life more than I want to be the next Martin Fowler (although I bet Martin has a kick-ass personal life too - that guy is amazing). Thinking about my career in terms of knowledge decay and accumulation though has changed my perspective a little.

First, I try to take the long view. I'm more wary of roles with excessively taxing expectations and few opportunities for novel experiences. I've seen quite a few colleagues take the bigger pay check at an employer where there'll be little opportunity to work with new things and learn. In 5 years, they realize that much of their valuable knowledge has evaporated and their pay is way out of whack with their actual worth. In some cases, I think making less money in the short term (at a better employer) will yield more money (and stability) over the course of a long career.

Second, given that time is limited, I try to invest most in knowledge that is durable. My energy is better spent accumulating knowledge that has a longer half-life - algorithms, application security, performance optimization, and architecture. Carving out niches in these areas, I hope, will better bullet-proof my career than learning the newest, flash-in-the-pan Javascript library.

In the end, perhaps I haven't really forged any new ground here, but it's been useful for me to think about my career in terms of these two things: knowledge decay and knowledge accumulation. I'd love to hear any thoughts you have!

Other Posts

• The 3 Motivational Forces of Developers
• The User Interface and the Halo Effect
• Pair Programming - My Personal Nightmare

The team goes through sprint planning, grooms the work, plays their little poker games, yada yada, and eventually gets to a set of work items that it endeavors to accomplish in the upcoming sprint. Great. Now what? Specifically, how do those work items get into the hands of individual developers? In other words, how does the work get assigned?

The simple solution is "central planning", and though this fundamentally contradicts the self-organization edict of Agile, it is, in my experience, the model which many teams use. The project manager/scrum master/tech lead has some intuition as to which developers would be best suited for which work, and goes about assigning stories/tasks based on that understanding. The end goal here is expedience - rather than a long drawn out discussion, let one person make the judgement call on who is the optimal person to complete a given piece of work.

The problem, of course, is that while this may optimize what's good for the project, it doesn't necessarily satisfy the goals of the individuals. A team may plod along with the "central" model for a few sprints, but it won't be long until some tension and resentment foments: "why does Jane get all the fun UI work?" or "I'm so f-ing sick of working with the accounting module", etc. Basically, we developers all want some degree of autonomy and self-direction, and so we'd like some choice in what we work on.

And this of course is why one of the pillars of Agile is self-organization. The problem is that there's no good, thoughtfully specified process for assigning work items, and so teams can instead end up adopting what amounts to a Hunger Games-style free-for-all. Planning is over, the work is ready, the horn blows, and the team rushes to their laptops to grab the work they want. This is probably an exaggeration - maybe instead there's a post-planning discussion where developers make dispassionate arguments as to why they should take the fun, new feature and someone else would be better off making a small enhancement to the registration form. In any case, the same tension and resentment can build, but this time directed at each other rather than the lead.

A solution that we've stumbled on (and have now used on a few projects) is the fantasy draft. At the risk of explaining the obvious, here's how it works:

• Sprint planning happens. Work for the sprint is identified, prioritized, and estimated (or not).
  
  
• Each person on the team picks a number out of a hat, bag, or container of preference.
  
  
• The team comes together for the draft.
  
  
• Starting with the first pick (duh), work items are chosen one-by-one until there is nothing left.

There are subtle refinements to this process, of course. We've instituted a clock (2 minutes per pick!), a looping scheme (2nd round goes in reverse from last pick to first), and a carry-over rule (work unfinished in a previous sprint is automatically carried over to the same developer). We've also considered "franchise players" and allowed draft pick trades.

In general, it works pretty well, and, if nothing else, is a lot of fun. Draft day comes with all the frivolity that you'd expect from a fantasy draft - someone wasted their first round pick on a trivial user story, someone couldn't pick in time, etc.

Beyond the fun though, the draft system has some other nice qualities, most notably fairness. People are in control of their own destinies. If they get the shaft this sprint with a shitty draft pick, well, next sprint they'll likely do better. It also distributes the "fun" work more equally. If the UI is the most desirable tier to work in, with the draft system it's more likely that most people will get an opportunity to get their hands in there in some capacity. This might come at the price of efficiency, but definitely aids maintainability and knowledge dispersion.

There are, of course, some downsides. The most obvious is that it gives no preference to seniority or capability. That complex integration piece can be picked up by the a junior UI developer if he so chooses. On the flip side, the most senior developer might get stuck writing the 93rd login form of their career (and be none too excited about it). As a "senior"-type person, admittedly, this can suck.

Anyway, I write this less to advocate the fantasy draft as the model that you should use, but more as a way to find out what others do on their teams. How is work assigned? What are the pros and cons?

Thanks to Handerson Gomes for the pictures!

The team is re-architecting a core service, and is considering whether to go with design A or design B. Both have pros and cons (in terms of scalability, performance, etc.) and the team is uncertain which route to go. From prior experience, you have solid reasons to prefer design B, but as the new guy on the team, you decide to let the conversation play out before you chip in your 2 cents.

Fail! Wait...already? Yes. What many people don't realize is that early opinions matter incommensurately, and if you feel strongly about a given decision, you need to chime in first. The reason for this is called a cascade. To see how it works, consider again your team, and their individual preferences before discussion:

As you can see, before discussion, there's a 4-2 split among the team in favor of the design you prefer, design B. All things being equal then, you would expect the team to arrive at a consensus for this design, but in reality, this consensus is anything but assured - it depends highly on how the discussion plays out. Imagine the team goes around the room and one-by-one states their preference and then explains their reasons. Al and Bob are always the most out-spoken, so they go first:

Both Al and Bob, who also happen to be the most senior developers, go on the record supporting design A and then give their reasons (which are sensible, by the way). Now, it's Carlo's turn. His preference is for option B, but he also is new to the team, so he begins to reconsider. Despite the fact that Carlo has a good reason for option B, he wonders, "is it worth it to go against the grain here"? Not confident in his own opinion and fearing a little blow-back from Al and Bob, Carlo decides its safer to just go along with the consensus opinion and states a preference for option A. And here starts the cascade!

It's important to note that not only did Carlo switch his preference to option A, but he also withheld a valuable bit of knowledge that only he had in favor of option B - knowledge that could have been persuasive to others. In any case, by the time it gets to Dana, Fran, and you, it seems as though the proverbial ship has sailed. There is solid support for option A, and there's not much sense in talking it out further.

But wait...didn't the team, collectively, prefer option B? What happened here? Why did the group decision succumb to the preference of the few who spoke first? This is the power of the cascade - people go along with the observed actions of others, even when these actions contradict their own private beliefs. This can happen for a couple of reasons.

First, people naturally like to agree with each other. This is so much the case that when we do, we actually both end up feeling better about ourselves ("Oh, you like React? I like React! We both must be exceptionally brilliant people!"). When we happen to disagree, depending on the social context, there can be a cost - others might actually think worse of us for our dissenting opinions. This is real, although it depends highly on the team environment and on the group leaders (we all know tech leads who are more or less accepting of divergent opinions). When people defer to the preference of a person with more seniority, against their own preference, this is called a reputational cascade, and it happens often.

Second, as people consider the preferences of others, they begin to discount their own reasons for favoring or opposing a given decision. This is called an informational cascade. "Wow, 100 people liked this song? Maybe I'm wrong, it probably is good!". As the cascade rolls over people, it gets ever stronger, as people withhold not only their original preferences but also information that they uniquely held that would be relevant to others in making the decision. This happens in plenty of real-world instances (for example, juries), and to sometimes alarming effect.

The moral here is probably not exactly what the title suggests, although it is true - speaking first will likely give you more persuasive power. More constructively though, we should try to structure our team discussions with an awareness of cascades. Get all the information on the table before allowing people to state their preferences. And if junior people are worried about going against the grain, let them speak first, or just have everyone write down their preferences on a piece of paper, and tally the anonymous votes. I've done this often, and it can make a boring decision fun(ish).

Anyway, I'd be curious to hear what you have to say. Does this happen on your teams? Do the outspoken people on your team have inordinate sway over team decisions?

"So...what are you using?"

...at which point you respond with the newest, flashiest piece from your current stack:

"Ohhh...it's just basic Java/Spring on the back-end, but on the UI it's Angular (!)"

The purpose of the exchange is of course not to objectively describe a set of technologies used to solve a problem. Instead, we highlight the "cool" language/tool/framework in order to reveal something about our ourselves, our identity. We're not just average programmers. We use Angular (or whatever). We're in the "in crowd". In a sense, the technologies we use are like the clothes we wear - they speak to who we are, what we stand for, our worth. It's technology as fashion.

Now of course the analogy is far from perfect, but consider some parallels from this description of the fashion cycle by sociologist Herbert Blumer:

The elite class seeks to set itself apart by observable marks or insignia, such as distinctive forms of dress. However, members of the immediately subjacent classes adopt these insignia as a means of satisfying their striving to identify with a superior status.

They, in turn, are copied by members of classes beneath them. In this way, the distinguishing insignia of the elite class filter down through the class pyramid. In this process, however, the elite class loses these marks of separate identity. It is led, accordingly, to devise new distinguishing insignia which, again, are copied by the classes below, thus repeating the cycle.

In a nutshell, fashion, whether in clothing, music, or programming, is about class differentiation. The elites innovate. The masses catch up so they can be associated with the elites. As soon as they do, however, the elites are no longer "elite", and so they must break away and innovate again. The masses eventually follow, and the cycle continues, ad inifinitum.

Obviously, there's more to it in the case programming - our choice in technologies can't be as whimsical as our choice in hair style. Our decisions must be grounded in a strong base of pragmatism - i.e. what we use should actually work. Still though, I think we understand that there's always been a degree of fashion in our industry. Whether it's Angular today, or GWT, Rails, or JSPs of yesteryear, there's always been that "next best thing" and we've always scrambled to keep up.

Now, sure, we often jump the bandwagon out of earnest hope that we've finally found the silver bullet, or some close approximation. I think just as often, however, we jump on the bandwagon so that we can sit alongside the "cool kids" and not be "left behind". Ok, we could have used a technology we know and are proficient with, but that technology is old and lame now. We use what's new. We're up on things. We're "current".

To be clear, it's not the innovation itself that is dangerous, but rather the vacuous emulation of it. When we use a tool, a framework, or a language only because it's new or "hot", we are taking on non-trivial amounts of risk to our projects and our clients. We make our systems less maintainable and unnecessarily complex. Even when the new technology is by all standards "better", we should be sure that it is better for our project and organization and solves the particular problems that we need to solve.

Unfortunately though, I feel like this empty, reckless adoption of new technologies is more prevalent than ever. For example, I hear this kind of exchange alarmingly often lately - a programmer unabashedly suggesting technologies to another without even understanding the context:

"You know, you really should have used MongoDB"

Really? We only had 1MB of data and the rest of the organization was using MySQL. More importantly, we finished the project in scope, on time, and on budget!

Again, I'm not at all bemoaning the process or cycle of innovation itself. Innovation is our business. We create. We make things better. This is great. I do think, however, that the ever increasing pace of innovation, on all fronts (e.g.NoSQL databases, Javascript frameworks, Mobile toolkits, etc.), is playing into our hard-wired psychlogical desire to "keep up", and to ill effect. I know myself that I'm often so overwhelmed by how much new "stuff" there is out there, and moreover by the paucity of free time I have to reasonably evaluate it, that I often catch myself reverting to fashion-based decisions. That guy at the conference told me that Go was the next big programming language - I better start using it!

So as I get off my soap box, let me make one last plea. If you're in the elite, keep the innovation coming (obviously). Those in the masses, however, let's not be afraid to let a bandwagon or two pass us by. Let's be judicious about adopting innovation. The right tool for the right problem. In other words, let's not be slaves to fashion.

As always, let me know what you think!

The application is launched, and it's a big success. Users are happy. Unfortunately, there's little time to celebrate, because with success comes requests for new features and enhancements. And they're coming in fast. Fortunately, the development team is up for the challenge. They start churning out code, and the team does frequent releases to keep pace with the feature demand.

Things are going well for a while, but as the application grows, things get a little more complicated. The team finds that even with solid automated unit testing practices in place some new features end up breaking old features in quite unexpected ways. Regression bugs. Ugh. Users get frustrated, and so business owners do too.

To avoid these embarrassing snafus, the team decides to slow down, and implement a more rigorous regression testing practice. The problem is, this being the nascent stages of the application, regression testing is all manual, and as everyone knows manual testing is time consuming. Exorbitantly so.

After just a few releases of comprehensive manual regression tests, it becomes obvious that this is a big bottleneck. The business demands frequent releases, but there just isn't the QA capacity to keep up. The team has two choices. They can choose to slim down its testing scope so that regression tests can be completed more quickly, but of course this would introduce a greater likelihood of regression bugs. Alternatively, the team could choose to release less frequently so not to repeatedly incur the high cost of regression testing, but then users would have to wait longer for new features, and time-to-market opportunities would be missed. Neither choice is pleasant, and the business grows concerned.

It's here that someone introduces the idea of automating the UI regression tests. Eureka! By automating tests, the team can have it's cake and eat it too. The scope of the regression tests needn't be narrowed, since automated tests can execute faster than a human. Also, releases can be frequent, because automated tests can be run as often as needed. The graph above captures the promise of automated UI testing.

Note that the cost function for manual testing is linear - it costs a fixed X for each run of the test. The cost of automated testing on the other hand is a step-wise function: to run it once costs more than running it manually (because you have to write the test), but then every subsequent run costs nothing at all. Overall return on investment is achieved when the two lines intersect, and so the more often you run the tests, the more you save.

This is the promise at least, and both sides (for once!) are completely on board. The business owners love the idea - they can get features to market more quickly and with high quality, and still not have to hire a larger QA staff. The developeres love the idea as well - automation is basically a virtue of every good programmer, and it's always exciting to play with the cool testing framework du jour to boot. A win-win!

Sadly, as anyone who has experience with automated UI testing knows, it never quite works out this way. While some organizations do eventually achieve some ROI automating UI tests, quite often the effort is a fruitless boondoggle - the tests that result are either unmaintainable, ineffective, brittle, or all of the above. Time and money are sunk, and little value is really extracted. The problem, in my opinion, is not that automated UI testing is hopeless, but rather that the expectation that is set up front about automated testing's ROI, as captured in the graph above, is overly simplistic. In my experience, effective automation is more expensive to implement up-front, and still requires non-trivial "care and feeding" throughout its life. In other words, automated UI testing is no silver bullet.

In the rest of this post I'd like to explore some of the subtle or unexpected challenges in automated UI testing I've come across in my project experiences, and hopefully re-frame a better image of its expected ROI. Ok, here we go...

Test resiliency vs. brittleness

The graph above belies probably the most important insight about automated tests: they're not all equal. Tests can be written to be resilient, such that a failure of the test almost always indicates a failure of the feature. These are tests you can trust. Alternatively, they can be written to be brittle, such that trivial changes to the application or data cause the tests to fail. These are tests to be wary of. When they fail, they must be examined, re-run, and then either tweaked or re-written altogether (i.e. test death).

In practice, tests can fail for all sorts of reasons. The location or order of UI elements on the screen might change. A brittle test would break, but a resilient test would account for this. Data might be valid when the test was written, but may have changed when the test was run. A brittle test would break, but a resilient test would set up fresh data and tear it down after. There might be a network hiccup that causes the UI to be unresponsive. A brittle test would break, but a resilient test would recognize this and try again. And so on.

As expected, it takes much more time and skill to create a resilient test than a brittle one, but the resilient test will live longer. Given this distinction then, it's clear that the cost function of automated tests should be bifurcated. The line below shows how brittle tests are relatively cheap to write, but will die quickly. The line above shows how a resilient test is more expensive to write, but will last longer:

It's important to note that this picture is actually generous for brittle tests. In many cases they will die before they reach the point of intersection with manual testing (i.e. no ROI). The canonical example of a brittle test is a record-and-play script created, say, with a simple Selenium plugin with almost zero effort. For a variety of reasons (like the ones given above: UI order, data, network), this type of test will probably die before the week ends.

Strong vs. weak verification

Another important distinction to draw of tests is their verification effectiveness. Tests can be written to be extremely observant, or completely clueless. For example, it's not uncommon that a suite of automated UI tests might run, pass with 100% success, and then later that day a business owner will notice that all the text is lime green and the content areas are in random places. The application is obviously broken, and any human tester would notice it in a single glance, but the automated test was never written to verify the look-and-feel of the application. Given that the entire point of a test is to catch defects, this test failed its mission.

The point here is that tests can be written with weak or strong verification abilities. On one end of the spectrum, the test could wander through the application and just verify that there were no errors. This would be cheap to implement, but it might not give much confidence that the application is truly working. On the other end of the spectrum, the test could try to mimic a human tester's judgement, verifying anything and everything a human would look for. This, unfortunately, is not easy (if even possible). There is a choice to be made, therefore, about the type of tests that should be created, and the more effective a test is at spotting defects, the more costly that test will be to write.

Maintenance is never free

Even if you choose to write resilient tests, maintenance of them is still not free (as the horizontal line in the first graph suggests). Someone must monitor test runs, and when a test fails, that person must investigate. Resilient tests should only fail for valid reasons, but a human still needs to make sure. In these cases, the maintainer will most likely first try to re-create the test failure, either by re-running the test and watching it, or stepping through it manually. If it breaks again, great, a defect report can be written up. If not (as often happens), the maintainer needs to dig in and figure out why it would break once but not again. What do the application logs say at the time the test broke? What else was happening at that time? What did the data look like? There are plenty of variables, and it often takes a non-trivial amount of time to understand what happened. In a nutshell, maintaining automated tests is not free.

The automation infrastructure

Beyond examining test failures, someone also must be responsible for kicking them off, or better yet configuring them to run on some schedule (from a continuous integration server, say). The more robust the setup, however, the more technical complexity that's taken on. Managing this complexity can cost a non-trivial amount of time and effort. Tests can suck resources from machines, causing headaches (e.g. memory errors, etc.) for administrators. Tests can take a prohibitive amount of time to run and need to be forked/clustered. Tests can hit security blocks (e.g. tests need to mimic user behavior, and users often need to log in, but how does the test login though without storing someone's actual password?). Or finally, tests can depend on frameworks and libraries that must be upgraded. The point is that automated tests introduce a slew of technical challenges never encountered in the world of manual testing.

Making the app testable

A final often-overlooked cost is the time it takes to make the application itself testable. For an automated test to drive a screen, it must have some hook for the UI elements. For example, in order to "Enter 'John Doe' in the 'name' field", the test needs some way to find the 'name' field. Some approaches for locating UI Elements (e.g. using preceding label names, XPath) can be brittle, since small, seemingly innocuous changes to the UI can still confuse the test, and cause it to not be able to find the UI elements it needs to drive. When this happens, tests fail.

Resilient methods for referencing UI elements can sometimes require changes to the application itself (e.g. adding an "id" attribute to input elements, if the application is HTML based). In some instances, making these changes can require a significant effort. Again, this is a challenge avoided when testing manually.

Conclusion

In this post I painted a gloomy picture of automated UI testing, probably too gloomy. My experience has not been that automated UI testing is necessarily ineffective or a waste of time, but rather that the promise land of easy, cheap, human-quality testing via automation is a chimera. I've seen many teams go down the path of automated testing with rosy expectations, only to emerge 6 months later with hundreds of man-hours burned, a set of brittle tests with weak verification abilities, and only a handful of successes (i.e. defects prevented). In a future post, however, I'd like to correct this gloomy course and talk about situations in which automated UI testing can work, in my experience. Until then, I'd love to hear any comments you had, good or bad. Thanks!

I've always been fascinated with how subtle, unexpected forces like this shape our behavior. This is the world of pyschological priming, and I think its advocates would proffer the answer is "yes", features in our environment probably do matter. One of the more famous studies of priming goes something like this:

A participant is sent to a room where she reads a series of words. Clean, private, thirsty. She next walks down the hall to another room for the second part of the study. Unbeknownst to her, however, walking down the hall is the study - in particular, what is being measured is exactly how fast she does it. If she's primed with words that conform to an elderly stereotype, she'll walk slower than if primed with neutral words. For example, Florida, wrinkled, bingo. Reading those words subconsciously impels her to slow her gate.

Now there's a bit of a brouhaha over the reliability of some of these priming experiments, but there seems to be too much evidence to discount the phenomenon altogether - there's something to it. External stimuli do impact our behavior in ways that fly beneath our conscious radar.

So it made me curious, are there subtle signals lurking in my physical environment that make me better or worse at my job? Or, maybe, could I plant some that would help my fellow developers (and I)? Riffing off of Notre Dame's famous mantra, I taped this up in our stairwell...

As the football players do, we all religiously hit the sign as we walk in to our office. And to my surprise, I found that the lines of code per developer after being exposed to the sign increased by 27%!

Ok, I'm making this up.

I'm sure it didn't have any discernible effect; it's just a singular token. But what about an environment rife with positive encouragement. Those cheesy motivational posters, perhaps? If I put one up that shouts "Courage" at me, do I unknowingly feel more confident about refactoring some tangled piece of legacy code? Or if I read "Integrity", am I more thorough with my estimates? (It's not that ridiculous, actually. There was a study that found that a motivational poster placed at the bottom of a staircase and escalator doubled the use of the stair case.)

A little more serious, I wonder if perhaps some of the value of displaying our project management artifacts, like a Kanban board, in our common areas is that they serve as positive primes - as we attend to them, our subconscious quietly whispters "stay on task"..."keep moving"..."work".

Or what about just positive symbolism without words? Bold colors. Open space. Live plants. Would we associate any of these with energy (or whatever), and would these positive associations then trickle into our work? Basically, is there a ROI in those flashy, inpsiring Google-style workspaces?

I'm not entirely confident of the power of priming effects, but I would not be surprised that the environment we work in, and in particular the signs, symbols, diagrams, or signals, would have some subtle impact. Recently I've tried to be aware of this - leaving diagrams, etc. around to keep me focused. What do you think? Do features like these in your environment matter? Or would you write the same code anywhere you were?

After 15 years in industry, I've come to realize that the most defining quality of a developer is his source of motivation. It undergirds all of his decisions and interactions. It predicts what kind of code he'll write, what technologies he'll prefer, and even how he'll succeed on a given assignment. And it's often quite easy to peg for any given developer after just a few days of working with him or her.

There are three and only three sources of motivation (so say I!), and they can be visualized as a radar chart, like so:

At each vertex is the motivational force, and right smack in the center is the absence of any motivation at all (i.e. "don't give a crap"). You know who I'm talking about. Anyway, each developer has his or her own unique "shape", where the greater the area of his triangle, the more motivation. For instance, here's how I'd draw me, plus or minus:

Although a developer might be driven in some portion by all motivators, there is always one force that is most dominant. I'll explain each:

Business Motivated: Driven most by a desire to get things done for the customer, these developers are often the "favorites" of the business folks because they deliver functionality quickly without many questions. Need a new feature at the last minute? No problem. These guys have a can-do attitude (even when a little push-back might benefit the integrity of the system's architecture). In terms of code, they think more concretely, and aren't always the best at creating abstractions that support re-use or other non-functional goals. They just want to get 'er done and see a functional product. On every project, a few of these developers are crucial. They get things done, and don't make waves.

Technology Motivated: These developers love learning new things for its own sake. They're always eager to find the newest framework, language, or methodology and will take every opportunity to try it out on their current project. The library was just released last week and was written by 1 guy over a weekend? Let's give it a shot! These guys know all the trending technologies, and have probably dabbled with them over nights and weekends. They love to try things out, and understand which tools work best. On a greenfield project they thrive, but when the field turns "brown" and new code turns into legacy, they look for greener pastures, or possibly worse, look for ways to shoe-horn in technology even if it's to the detriment of the system.

Problem Motivated: Hard problems excite these developers, independent of whether what technology is employed or even it adds value for the business. It's all about the puzzle. Coming up with an elegant, clever, or quality solution is the victory. If it helps the business in the process, great (and often it does). These developers are interested in new technology insofar as it presents better ways of solving problems, but not so much interested in dabbling just to know what's out there. While their solutions are solid, sometimes the details slip.

Now of course this isn't the "right" classification system, but I've found that it is useful to think about my fellow developers as one of these 3 (or actually 4) types - the get 'er done's, the technologists, the problem solvers, and the don't-give-a-craps. An awareness of each of our differences is helpful. I know that each will have certain strengths and weaknesses, and will contribute to the project in different ways, and in different parts of the life-cycle.

For instance, when starting a new project, we look for the technologists to help inform us about what framework or library on the horizon could be most useful. Or when the team gets stuck in the weeds of over-analysis, we rely on the get 'er done's to just dig in and start delivering working code. And when a problem is particularly thorny and needs to be thought through, we look for the problem solvers to step back and find a way out of the swamp.

In other words, everyone contributes in a different way. Understanding people's underlying sources of motivation can help you get the most out of your team. It can also make you cognizant of keeping a team in balance - too many of any particular type is never healthy, and in fact can spell disaster for your project. Whereas a team of all technologists or problem solvers might obsess about the infrastructure and never deliver value to the business, a team of all get 'er done's might more quickly create a big ball of mud.

Anyway, let me know what you think! What type of developer are you? And does it matter?

Recent Posts

1. Expectations and the Endowment Effect
2. Consistency and Innovation: Pick One
3. Pair Programming - My Personal Nightmare

At the Summa Summit this year, we tried out a new, hands-on coding activity that we dubbed "Coding Spaces". The general idea was to allow our technical consultants to get down and dirty (geeky!) - dabble in some new technologies, collaborate and program with others, and maybe even build something cool. Topics included Angular, Scala, Clojure, LEAP Motion, Salesforce, and Common Lisp to name a few. It was a ton of fun. I led one session, an AI Challenge, and this post is a recap of what our group did in case you're interested, or in case you'd like to try this challenge on your own (the source is included).

The Challenge: Prisoners Dilemma Tournament

The challenge we competed in was a Prisoners Dilemma tournament, with a twist of Survivor. A little background is probably in order! The Prisoners Dilemma is a canonical game in game theory, which is typically used to explain why people don't cooperate with each other, even when it seems like it's in their best interests to do so. The game gets its name from this story:

You and your buddy get arrested for a crime that you both committed. It's clear that the authorities want you to confess, so they split you up into separate interrogation rooms, and interrogate you individually. If you confess and your buddy doesn't, the cops will go easy on you, say 1 year in jail, but they'll throw the book at your buddy, say 4 years in jail. The reverse situation is true as well - if he confesses and you don't, he gets 1 year and you get 4. If you both confess, however, they go a little easier on both of you, say 3 years. If neither of you confess, they give you 2 years on some trumped up charge.

Of course you know it'd be better to both "stay mum", but when you look at the outcomes in the table below, it's clear that it's in your best interests to confess no matter what your buddy does in the other room. If he stays silent, you're better off confessing (1 year

 

 

In the 1980s, political scientist Robert Axelrod organized a tournament where participants of different backgrounds (mathematics, psychology, economics, etc.) created programs to play an iterated prisoners dilemma (i.e. round robin, multiple games) against each other. To everyone's surprise, the algorithm that won was the simplest of all, a "tit-for-tat" program devised by mathematician Anatole Rapaport. This tournament is often cited as a defense of cooperation in the large. Even though in small groups or in isolated instances being "mean" (in game theory parlance, "defecting") is the better strategy, over time it's much better to be "nice" (i.e. "cooperate").

This is a obviously a blazingly fast tour of the Prisoners Dilemma game, and I'm sure I didn't do it justice. I had read about this tournament initially in a book called the Origin of Virtue, and then in Axelrod's own The Evolution of Cooperation a while back, and had always been fascinated by it. Is it really the case that tit-for-tat wins out? How long does it take for being "nice" to pay off? What if you're the only "nice" person (program!) in a world of "meanies"?

Well, this is what we endeavored to find out in our Coding Spaces. We replicated the famous Axelrod tournament, and as you'll see, got some very interesting results!

Game Setup

I created a simple engine to pit different game playing agents against each other. All the participant needed to do was implement the following interface (in any JVM language - I used Groovy). If you want to follow along, you can download the source from GitHub here.

  package com.summa.summit.spd

  public interface Player { 

     /** Called at the beginning of each round */
     void onNewRound(int numGames)

     /** Called when your about to play a new opponent */
     void onNewOpponent(String opponentId)

     // Choose to cooperate or defect against the other player
     Decision play()

     /** Called after play() is called, and tells what your opponent did */
     void onGamePlayed(Decision opponentsDecision)

     /** If playing survivor mode, at the end of the round vote one player off */
     String voteOffIsland()

     /** Return the name of your program */
     String getName()

  }

Note that the Decision returned by the play() method is just a simple enum with two values: Cooperate and Defect. To give an example of a simple player, a very naive, always-be-"nice" strategy could look like this (in Groovy):

  package com.summa.summit.spd

  public class MrNiceGuy implements Player { 

    def opponents = [:]
    def currentOpponent = ""

    Decision play() {
      return Decision.Cooperate // always be nice!
    }

    String voteOffIsland() {
      return opponents.isEmpty() ? currentOpponent : opponents.max { it.value }.key
    }

    String getName() {
      return "Mr. Nice Guy"
    }

    void onGamePlayed(Decision opponentsDecision) {
      if(opponentsDecision == Decision.Defect) {
        opponents[currentOpponent]++ // vote off the meanest person
      }
    }

    void onNewOpponent(String opponentName) {
      currentOpponent = opponentName
      opponents << [(currentOpponent) : 0]
    }

    void onNewRound(int numGames) {
      currentOpponent = ""
      opponents = [:]
    }

  }

Tournament

Out of the box, the initial Prisoners Dilemma game has four built-in players...

 

If you're so inclined, once you download the game engine from GitHub (here), you could play just these guys against each other by running the following from the root directory:

> java -cp ./dist/groovy.jar;./spd.jar com.summa.summit.spd.SurvivorPrisonersDilemma

This should give you a main menu, like the following:

MAIN MENU
-----------------------------
1) View players
2) Add players
3) Remove player
4) Set # games per round (current = 10)
5) Set survivor mode (current = false)
6) Play!
7) Exit

Ok, back to the recap. On top of the default players, we had 7 "real" participants. Their strategies were, roughly:

 

Again, if you're so inclined, you can just run the following (on a Windows machine, sorry!) from the root to play all these players against each other:

> spd.bat


Results

When you run the tournament in basic (non-survivor) mode, what you'll find is that with low number of games (~100), the mostly or purely "evil" strategies always do the best:

SAY WHAT AGAIN!: 2,482
SpaghettiCoder: 2,484
Joe Evil: 2,486

It's only when you increase the number of games past about 1,000 that Sharbaugh, a opportunistic cooperator, begins to take the lead. Tagging not far behind, however, are still the evil strategies

SAY WHAT AGAIN!: 24,596
Joe Evil: 24,606
SpaghettiCoder: 24,628
Sharbaugh: 24,855

There are a few interesting observations to be gleaned here.

First, the Tit-for-Tat strategies never do well because they can't establish trust with anyone. SummaNoid 3000, a pure Tit-for-Tat wants to cooperate with Frank Dux, but as soon as Frank Dux randomly defects, SummaNoid 3000 never forgives, and they end up defecting against each other the rest of the way and miss out on the potential benefit of cooperation.

Meanwhile, the evil strategies thrive by preying on the blissfully, ignorant players (Mr. Nice Guy and Bob Backstabber). When you take these players out of the mix, however, the mean strategies fall to the bottom, and the cooperators like Sharbaugh, Frank Dux, and SummaNoid 3000 float to the top. Here's a sample 1,000-game round without the "dumb" strategies:

Rupert Random: 13,459
El Santo: 15,012
Adamo: 15,014
SAY WHAT AGAIN!: 1,5014
SpaghettiCoder: 15,020
Frank Dux: 15,041
SummaNoid 3000: 15,157
Sharbaugh: 15,443

In other words, when others are very nice but not very smart, it's easy to get away with being "mean"!

There was also an optional survivor mode component to the game, where at the end of each round each player could vote one other player "off the island". Strategies differed as to whom to vote off. Some voted off the "meanest", some the "nicest", and some whomever did best against them. Because many players had non-deterministic (random) strategies, there didn't seem to be any predictability in these games. Although El Santo and Sharbaugh seem to win more often (from what it seems), other players also win from time to time. It's not clear if there's a conclusion to draw from the survivor mode.

In the end, we could analyze and play this game for hours/days/weeks more, but alas the fun had to end, and we had to get back to real work. It was a ton of fun, and hopefully we'll do it again soon. Let me know if you want to use the source code here, or just share an AI challenge that you've played.

You're just a few days into a sprint. Your customer, a reasonable enough guy, identifies a new feature related to something your working on. It's small, maybe just a 3 story pointer in a 60 point sprint, but you have a good amount on your plate as it is.

Since you aim to please, however, you let the customer know that you'll give it a shot. If things are tight at the end of the sprint though, it'll need to get pulled from scope. No big deal.

Sure enough, the end of the sprint comes, and you're swamped. You can finish everything you originally committed to, but you won't have time to finish the extra feature. You let the customer know, thinking it'll be a non-issue. To your surprise though, he's actually kind of torqued. But you're delivering what you originally committed to!

What happened here? You never truly committed to the feature in the first place, it was really just a bonus. Why would the customer be upset?

At least part of the explanation comes from prospect theory, and it's something called the endowment effect. In a nutshell, the endowment effect asserts first that we hate losses more we than enjoy gains, and second that both losses and gains are calculated not absolutely, but relative to some reference point. What does all this mean? Let's look back at the example.

Initially, the business owner expects 60 story points to be delivered, and he's content with this. A few days into the sprint, when you tentatively commit to the new feature, you do indeed make him happy. To put it in numbers, say...

3 story point gain = 10 "happiness points"

At this point, however, his reference point shifts. Whereas he was initially expecting 60 story points to be delivered, his mind is now attached to 63 points.

Fast forward to the the end of the sprint. You realize you can't actually deliver the extra feature, and so you take it out of scope. You assume the customer is thinking in absolute terms - he started out expecting 60 story points to be delivered, and that's what he's getting. This is where you're wrong.

The customer at the end of the sprint is only thinking from his current reference point, which is 63 story points, and so he perceives you not completing the new feature as a 3 point loss. Further, since people dislike losses more than they like gains, that 3 point loss doesn't just "cancel out" the 3 point gain from before, but it actually looms larger. Again, in numbers, prospect theory explains that he might feel the loss like...

3 story point loss = -20 "happiness points"

In other words, because you "gave" the customer the new feature, and then took it away, he is less happy than if you never tried in the first place! Rationally speaking it doesn't seem like it should make a difference to him (+3 story points - 3 story points = 0!). Given how our cognitive apparatus perceives losses, however, it does!

This effect shows up all over. Paraphrasing an example from the famous economist Daniel Kahneman, imagine your favorite football team is in the playoffs. You'd be willing to pay $300 to go, and fortuitously you find a stranger to sell you a ticket for that much. A few days later, the game is getting a ton of hype, and you learn that tickets are going for $1,000 online. Do you sell your ticket? No! Even though you valued the getting of the ticket at $300, once you have the ticket, you value the losing of the ticket at more than $1,000. In other words, once you have something, you hate to lose it.

What does this mean for us in software development? Well, as developers we need to be conscious of our customers' inherent loss aversion, and so always be cautious when setting reference points of any kind. This is not just limited to scope issues, either. If you tell the customer, for example, that you'll be done with some task in 5 days, you've just set the reference point. Finishing in 4 days (a gain of 1 day) will make the customer happy, but not as unhappy as will finishing in 6 days (a loss of 1 day). Losses rule!

In the end, it boils down to the advice a colleague gave me on one of my first projects: "under-promise and over-deliver". With only our words, we define the expectations others will use to measure us by. Understanding loss aversion and the endowment effect, we can use this to our advantage!

Recent Posts

1. Consistency and Innovation: Pick One
2. Pair Programming - My Personal Nightmare
3. The User Interface and the Halo Effect

It can all start with a simple text box...

TextBox textBox = new TextBox();

TextBox textBox = new TextBox();
textBox.setStyle("txt-standard");
Label lbl = new Label("First Name:");

TextBox textBox = new AcmeTextBox("First Name:", "txt-standard");

And there it is. This trivial example captures the essence of a problem that occurs over and over on every project, large and small, team or solo. The problem is: what do we do when we find a better way of doing things? Note that "better" can mean not just more maintainable, but more extensible, scalable, faster, etc. Of course the decision seems obvious (go with the better solution!), but it's not always that easy. Just consider your choices for the TextBox example:

1. Refactor: Go back to each TextBox initialization, and replace with the AcmeTextBox.

2. Live and Let Live. Use the AcmeTextBox on all new screens, but leave the old code as is.

3. Stay the Course: Stick with the old, clunkier TextBox initialization.

The lead developer will argue: if it's genuinely a better way of doing things, then let's go back and refactor the old for the new. Not only is it a better solution, but there's no worry that someone will inadvertently copy-and-paste from the old code, and propagate the inferior solution. (Option 1: innovation + consistency)

But "not so fast", the PM will say. If the screens functioned correctly before, don't touch that code! Regression testing costs time and money, and there are deadlines to meet. If you want the better solution moving forward, great, but let sleeping dogs lie. (Option 2: innovation + inconsistency)

Enter the curmudgeon developer: it'll just be confusing to do the same thing in two different ways, so let's just deal with a little extra typing and keep things consistent. (Option 3: no innovation + consistency)

So who's right?

The PM and the curmudgeon developer have valid points, there are costs to change, but if we heed their advice too often, it's a quick slide down the slippery slope into architectural mud. A codebase rife with inconsistency is a maintenance nightmare. So too is a codebase that never evolves.

In our gut, we want to listen to the tech lead, and always go back and refactor, but we realize this is a tough sell for all stakeholders. How do we make the case then? Well, we must show not only that it's worthwhile to implement the new solution:

Cost of Solution < Value of Solution_{(new code)}

...but also that there's benefit to go back and refactor the old code. In other words:

Cost of Solution + Cost of Refactoring < Value of Solution_(new) + Value of Solution_(old)

The problem is, that while the cost side of the equation is usually a pretty tangible number incurred in the short term (e.g. it'll cost 20 hours to refactor all instances of TextBox to AcmeTextBox), the value side of the equation is neither easy to quantify nor realized in any near-term horizon (e.g. how much productivity is actually gained by using AcmeTextBox everywhere? And how do you convey this to the PM or QA lead?). Getting all stakeholders to buy into refactoring, therefore, is a non-trivial leap of faith: it must pay for something now in hopes of getting some return down the road.

What can we do? I think we need to address both sides of the equation.

On the left side of the equation, we need to to work vigilantly to reduce the cost of refactoring, by committing to continuous integration, unit tests, etc.. If it's cheap to refactor, then there's less of a burden to show that consistency is worthwhile.

On the right side of the equation, we can do a few things. First, we can try to use data to help quantify the value of consistency (i.e. maintainability) using tools like Sonar or Cruft4J, and also concepts like technical debt. If stakeholders understand the value, it'll be easier to weigh against the costs of refactoring. Second, we need to build trust by continuously delivering value to the business side, ensuring that refactoring expeditions are not done to the exclusion of tangible functionality. Philippe Kruchten presented an interesting approach for doing this using a color coded backlog.

In the end, the approach of going-back-to-implement-the-new-solution-everywhere won't always win the day. The cost of refactoring will be too high, or the value of consistency won't be appreciated. In these cases, you'll be stuck with the age-old software development dilemma: consistency or innovation. I usually side with the latter, but I'd love to hear your thoughts!

Recent Posts

1. Pair Programming - My Personal Nightmare
2. The User Interface and the Halo Effect
3. Velocity and Story Points - They don't add up!

Those who reject pair programming are assumed to be cowboys, slackers, or social recluses. Well, I'm none of these (at least I don't think), but yet I still hate the idea of Pair Programming. For what it's worth, here's why...

We are no longer a culture that respects silence. The extroverts have won. Everything must be done collaboratively. Everyone must be available to everyone, all the time. There should be no more personal space, and no ownership of work. Basically, we believe that two heads are always better than one.

And yet it should be obvious to us that this is not always the case. Just in the world of programming, some great works of innovation and craftsmanship have sprung not from a team or a pair, but from the efforts of one person. I think about Ant, a huge leap forward for the Java community at the time, developed by one guy on a flight from Europe to the US (1). Or a tad more current, consider Notch of Minecraft, Marco Arment of Instapaper, or Gabriel Weinberg of DuckDuckGo: all solo achievements (2). In fact, one of the most influential programmers (if not people) in the world, Steve Wozniak, famously advocates:

"Work alone. Not on a committee. Not on a team."

Going further, some of the greatest thinkers in science and art are of the heads-down (i.e. introvert) variety - think Darwin, Einstein, Newton, or heck, even Dr. Seuss (3). Even John Steinbeck chips in here:

Nothing was ever created by two men. There are no good collaborations, whether in music, in art, in poetry, in mathematics, in philosophy. Once the miracle of creation has taken place, the group can build and extend it, but the group never invents anything. The preciousness lies in the lonely mind of a man. (4)

Ok, I'm waxing philosophic here, but back to our own little corner of the world, software development, why would we believe, as some argue, that hyper-collaboration (e.g. pair programming) is a precondition to quality or productivity when we can readily think of so many counterfactuals? Why, for some, is Pair Programming a "must do", all the time?

I believe it's just a reflection of one's own personal psychology. Simply: some people enjoy this style of work, and so they prescribe it, vociferously, for everyone.

The truth is, however, that one third of us are introverts (and probably more for programmers!). In general, we not only prefer but thrive when working in solitude. It's not that we don't like people, it's that our brains are wired to be more disrupted by stimulation (and, for good or bad, this includes pairing). For us, doing quality work is about getting and staying in "the zone". When we do, we're highly productive. When we don't, we flail.

Demarco and Lister, in their famous Coding War Games experiment, demonstrated just this - the best predictor of quality for programmers, they found, was not years of experience or salary but rather how quiet their office environment was.

This used to be a respected insight. In fact, #8 on the Joel on Software Test of good places to work is "Do programmers have quiet working conditions?" Sadly, however, the culture of hyper-collaboration has rolled over our better sensibilities, and to be perfectly honest, I think it kind of sucks.

Pair Programming, an extension of this "everything-together" culture, has permeated our thinking to the extent that many think that one person working in isolation is not just ineffective, but boring to boot. For me, it's the opposite. My best work is done in solitude, and the state of flow is what I enjoy most about being a programmer. It's not about being a "cowboy", or thinking I'm above mistakes. I'm a huge advocate of rigorous code reviews, and I benefit daily from the insights of others. It's just that the hyper-collaborative state of pair programming doesn't make me a better (or happier) programmer. Please take my word for it.

When people describe pair programming as a practice that they benefit from, cool, I get that. But when they take the next leap and advocate (or mandate) the practice for me, because they "know" I'll benefit from it (and have "data" to prove it!), slow down. The methods by which people produce quality work are as varied as we are. Consider some great achievements in the world (or just on your project!) and this should be obvious. To claim that this extrovert's ideal of pair programming is a "best practice" for all is foolish, I don't care what the agile dogmatists say.

References

1. Ant: The Definitive Guide, Steve Holzner
2. Pair Programming considered harmful?, John Evans
3. Quiet: The Power of Introverts in a World That Can't Stop Talking, Susan Cain
4. East of Eden, John Steinbeck

Recent Posts

1. The User Interface and the Halo Effect
2. Velocity and Story Points - They don't add up!
3. A Room Full of Techies

Unfortunately, as we know, it's just not that easy. Cleaning up code takes work, and every hour of cleaning up code is an hour of not adding functionality. This makes Project Managers and Business Owners squirm (justifiably!). Additionally, cleaning up code is also risky, since a line of code changed is also a line of code that could break. And this, of course, makes QA nervous.

The point is that although cleaning up code is inherently a developer activity, it requires team buy-in. And to get this buy-in, you need to prove to Project Managers, Business Owners, and QA that it's a worthwhile endeavor. Even if they accept the concept of technical debt in principle, before they "ok" your code cleaning expedition they probably still want to know:

• How big is the problem (i.e. how does our code quality compare to other systems)?

• Are things getting better or worse?

Armed with just a long list of obscure coding violations from PMD and the others, these questions are difficult to answer. Sure, your system has 1,358 violations, but is that good or bad? Are these violations even important?

This is where a tool called Cruft4J can help. Cruft4J is a static source code analysis tool, like the others, but instead of a detailed list of coding violations, Cruft4J spits out just one score:

To answer the first question, how big the problem is, running Cruft4J once will give you a score which you can compare against some benchmarks of open source code. Obviously, the cruft score isn't a perfect measure of code quality, but if your system scored a 200, and the average score is 52, you know something is probably wrong.

To answer the second question, are things getting better or worse, running Cruft4J over time (or better yet, hooking it into your build) makes it easy to spot trends over time, so you can understand whether the ship is headed in the right or wrong direction.

Now there are other tools, like Crap4J and Sonar's Technical Debt, that too provide a single score. These tools, however, include factors that, in my opinion, are sometimes dubious indicators of code quality. For instance, Sonar's TD considers the percentage of methods that have Javadoc - but just because a method has a Javadoc comment doesn't mean that Javadoc comment is actually useful! Cruft4J, by contrast, stands on the unassailable principles of Decomposition and Don't Repeat Yourself. It's never the case that a complex method could not be broken down, or that copy-and-pasting code is preferable to other forms of re-use.

In the end, Cruft4J is not the solution, but instead a vehicle for getting the conversation started. As Business Owners and Managers understand the extent of the code rot, a better case can be made to not only clean code up, but to invest in processes that prevent bad code in the first place (e.g. code reviews, unit testing, etc.). Anyway, I hope it helps, and I'd love to hear your thoughts.

A final note: Cruft4J was built as part of an internal initiative we call Summa Labs. In Summa Labs, we work on side projects, big or small, for the purpose of honing our technical skills, collaborating with others, building cool stuff, or just having fun. If you're interested in Summa or Summa Labs, contact us!

You've been working feverishly to implement a new feature for your customer. There's some crazy complexity in the business logic, and a very simple UI as well. It's a lot of work.

With a herculean effort, you just about meet your deadline. The business logic is bullet-proof, but the UI is still a little rough. No worries though, that's the easiest and least risky part.

You head off to demo it for the customer. They immediately start picking at the UI: "the alignment of this column is off", "this label should be bold", and so on. Before they even get to the actual functionality (the meat of the work!), you can tell they're already a little underwhelmed.

In the end, everything works great, just a few minor UI glitches. You know it's truly a big success that it's even functional, but the customer isn't ecstatic, and you can't help but feel a little dejected.

So is it reasonable to feel this way? Yeah, probably. But it shouldn't be unexpected. As most seasoned developers know, the user interface carries an incommensurate amount of weight in terms of the customer's perception of the quality and completeness of the product. One reason this is true is due to something called the Halo Effect.

In a nutshell, the Halo Effect is a built-in cognitive bias that drives us to believe that if our initial impression of something (or someone) was favorable, the rest will be favorable as well. For example, plenty of studies have shown that we're more likely to confer positive traits (like intelligence, creativity, etc.) to good looking people, even when we know little (or nothing!) about them other than their appearance.

Or consider another example adapted from a famous experiment by Solomon Asch: imagine there are two people vying for a job, and all you know are a few attributes. Person A is...

intelligent, inquisitive, calm, serious, passive, unambitious

...and Person B is...

unambitious, passive, serious, calm, inquisitive, intelligent

Without thinking too deeply, who would you hire? Your intuition probably steered you to Person A, but if you look closely, both A and B have the same exact attributes, just in opposite orders. Why would we perceive them differently?

Well, after reading just one or two characteristics, we already begin to form a general impression of the person. Subsequent characteristics can serve to enrich this view if they are consistent with the initial characteristics, but if they conflict then they get filtered out (or at least carry less weight). In other words, our brains prefer coherence, and disregard dissonance.

Back to the world of software development, it's now more clear why minor user interface peccadillos (a typo in a label, for example) can significantly impact a customer's perception of the overall quality or even value of the software we build, to an extent that seems capricious or unfair. That's just how our brains work.

Essentially, the user interface is the physical appearance of our applications, and as such, customers will use it to form their initial impressions of the application as a whole. Once this impression is formed, it's difficult to undo. If the user interface is organized, clean, and pleasing, the customer will naturally assume the system behind it is too; but the opposite is true as well.

In the end, especially if you're early on with a customer and trust hasn't been completely established, don't focus on the back-end to the exclusion of the front-end, even if the UI is a cinch. That mis-aligned column may only take 15 seconds to fix, but it'll be worth a whole lot more in terms of your customer's perception of quality!

In practice, however, I believe any calculation of velocity (based in story points) is doomed to be dangerously inaccurate and misleading for either the short or long term. Here's why...

The first problem is that story points are not additive. Mike Cohn touches on (but does not fully address) this in a recent post. Using his example, imagine a team that uses story point buckets of 1, 2, 3, 5, and 8. Further, consider the median number of hours to complete user stories of each size:

Now assume your team's velocity is determined to be 16 points. It would seem that you could pluck any combination of user stories off the backlog (according to the business owner's prioritization, of course!), and be fairly confident that you could complete this work so long as the stories all sum to 16 points. It's easy to illustrate, however, why this doesn't work. For example, take just two different combinations of stories:

Note that although the story points both sum to 16, the actual hours differ significantly: 222 hours for the first combination of stories and 362 for the second. This is not a negligible amount! It could very realistically cause a delay in a release (or alternatively, a copious amount of slack time for the development team). Either way, something is wrong.

The root problem here is in the correspondence of story points to actual hours. To be able to calculate velocity, it must be the case that story points not only preserve a relative ordering, but their numeric ratios as well. In other words, it must be true that two 1 point stories equals the same amount of effort as one 2 point story. Unfortunately, story points in practice are little more than ordinal values - e.g. we can rely on a 2 point story being bigger than a 1 point story, but not necessarily a 2 pointer being twice as big as a 1 pointer.

In general, the worse this correspondence between story points and actual hours is on your project, the more unreliable your velocity will be. To ensure that the correspondence is consistent, therefore, a project manager must be vigilant about calculating these statistics (as in the table above) and then presenting them to the team, so that the team can best adjust their estimates (for example, using the data above, if the team wanted to keep its notion of a "1", then it would need to adjust down its assessment of a "2").

Assume then that the correspondence between story points and actual hours is perfect. There's still a second problem: teams very seldom track actual hours. Instead, "actuals", as in the example above, are most often derived from the hour-based task estimates given during the planning phase to tasks. For example, a simple "Add order" user story might be broken into three tasks, and each of these tasks would be given hour estimates prior to starting the sprint.

• Implement UI (24 hours)
• Implement business logic (10 hours)
• Test (4 hours)

Summing all these tasks, the project manager would get the total number of hours for that story, which is then used to burn down from. The problem is, of course, these hour estimates seldom map to how long these tasks actually took, and so using these "actual hours" to measure the correspondence of story points to actual hours will be spurious. Again, velocity is chimerical.

Finally, a third problem, as a number of experts have pointed out, is that a team's assessment of story points adjust over time. For example, what the team used to think of as a "2" a few months back may now be considered a "3". This type of adjustment is fine, again, if story points are only to be used as relative values, but if they are used to calculate velocity, it simply destroys any hope for accuracy, because it is simply not mathematically valid to add them together.

As an analogy, if half-way through your sophomore year your college switched from a 4 point scale to a 5 point scale, they obviously couldn't calculate your GPA by just adding all your grades together and dividing by the number of units. The scale changed! Again, the point is that because story points aren't additive, velocity just doesn't work.

In the end, I completely understand that story points and velocity were designed specifically to be a low-ceremony estimates, and further that the purpose of agile is to spend more time delivering working software and less time on producing plans for delivering software. That's great. However, there seems to be a very pervasive and dogmatic belief in the robustness of velocity, and it's just not empirically warranted. The idea is great in concept, but it's based on a the faulty premise: that story points can be used like numbers.

Personally, I severely doubt that velocity can be used reliably for long term forecasting, and definitely not for sprint planning. Just my thoughts - what do you think?

  DateTimeFormat f = DateTimeFormat.getFormat("MMM dd yyyy");
  lblDate.setText(f.format(user.getBirthDate());

After testing this feature locally, everything works as expected, and so you ship it. Voila! A few days later, however, the help desk calls and says that a user is complaining that their birth date is one day off. You assume it must be a data issue, so you check the database. The user’s birth date is:

1977-06-02 00:00:00.0

You then check for yourself within the application, and the users birthdate shows up as June 2nd, as it should. However, the user claims the apps says his birthdate is June 1st, and he has a screen shot to prove it. What the heck?

Well, upon further digging, you find that the user is in California, 3 time zones away from you, in New York. Seems like maybe this is the problem. You throw a few debug statements in your app, set the Time Zone on your local machine to Pacific, and sure enough, you recreate the problem. GWT logs that the user’s birth date is:

06/01/1977 21:00 -0700

In other words, 9pm the day before. What’s happening is that GWT by default uses the client’s local time zone when formatting dates with DateTimeFormat, so if a date is stored as midnight in the database for a given time zone (EST, in this case), for any time zone after the birth date will display as the day before.

Part of the root problem is that that storing a birth date as a Java Date is really overkill for most cases, since often all that’s needed is the day, month, and year. What the Java Date class gives us, instead, is some number of milliseconds relative to 1970 (plus if after, minus if before). On the server side everything would be fine, since formatting this date would use the timezone of the server, which is what the date is stored in. On the client side, however, GWT will use the timezone of the client machine, and so there could be a discrepancy.

Ok, back to the scenario. Armed with this knowledge, you then decide to hard-code the TimeZone when calling DateTimeFormat.format(). Knowing that EST is -5:00 hours from UTC/GMT, you would pass +300 minutes as an offset (note: it would seem more intuitive to pass -300, but this is not the case). So…

  DateTimeFormat f = DateTimeFormat.getFormat("MMM dd yyyy");
  TimeZone est = TimeZone.createTimeZone(+300);
  lblDate.setText(f.format(user.getBirthDate(), est);

Now, you test the change locally, this time checking for different time zones, and again feel confident that the bug is slayed and you ship the code to production. A few months later this time, you’d get another call from the help desk with the same exact problem: the user complains that his birth date is displaying one day prior to his actual birth date. Argh. What changed?

Thinking about it, you realize that what’s different is the season! A few months ago it was summer, but now it’s winter. Daylight Savings Time! What you didn’t account for is that user was born in June, which is really only -4:00 hours away from UTC/GMT, but you’re still offsetting -5:00 hours, and so on the client side the user’s birthdate is really:

06/01/1977 23:00 -0500

What you need is to take into account Daylight Savings Time in your format method:

  DateTimeFormat f = DateTimeFormat.getFormat("MMM dd yyyy");
  TimeZoneConstants t = (TimeZoneConstants) GWT
     .create(TimeZoneConstants.class)
  TimeZone est = TimeZone.createTimeZone(
     t.americaNewYork());
  int offset = est.isDaylightTime(date) ? +240 : +300;
  TimeZone tz = TimeZone.createTimeZone(offset);
  lblBirthDate.setText(f.format(date, tz));

Now, you think you’d be out of the woods, but unfortunately not. A few months later, you’d get yet another call from the help desk about the same freaking problem. You’re ready to lose it. What’s going on this time? Well, this user, you realize, is older than the others. He was born on:

1957-08-22 00:00:00.0

…but again his birthdate was displaying as the day before. Well, it turns out that GWT’s TimeZoneConstants class doesn’t consider Daylight Savings Time prior to 1970, and so the isDaylightTime() always returns false, and the -5:00 hour offset is always used, causing the same exact problem.

At this point, there are a few viable solutions. First, you could just format the date on the server side and pass it to the client as a String. Alternatively, you could create your own custom Date class (either as a subclass to Java’s Date or otherwise) that either takes into account TimeZones or just codifies dates as day/month/year. There are a few other possible solutions, but one of these two might do the trick.

Anyway, I hope this post has helped you avoid this problem. I’d love to hear any strategies you’ve come up with. Thanks!

Starting with the client tier and working my way to the server, here we go...

1. Handle all uncaught client-side exceptions

The first thing and simplest thing to do is to ensure that all unhandled exceptions bubbling out of the client tier are caught. You can accomplish this by registering an UncaughtExceptionHandler, like this:

  public class Example implements EntryPoint {
    public void onModuleLoad() {
      GWT.setUncaughtExceptionHandler(new
        GWT.UncaughtExceptionHandler() {
        public void onUncaughtException(Throwable e) {
          // do exception handling stuf
      }
      // do module loading stuff
    }
}

Pretty straight-forward. What happens if you don't set the UncaughtExceptionHandler in your application? Well, in hosted mode, the default handler will conveniently log the exception and stack trace to the console. In web mode, however, the handler is null, and so exceptions "escape" to your browser, which is likely not the user experience you're looking for (though it does provide the opportunity to use a Javascript debugger for further diagnosis).

Also, be aware that your custom UncaughtExceptionHandler only takes effect after the onModuleLoad() returns. In other words, if there was an exception raised where the comment "do module loading stuff" is in the example above, this exception would not be handled using your custom UncaughtExceptionHandler that you just defined. This is a known issue, and will not be resolved. As a work-around, if you expect exceptions could be thrown within onModuleLoads(), then you can extract the guts of onModuleLoad() to another method, and invoke it using a DeferredCommand:

public class Example implements EntryPoint {
  public void onModuleLoad() {
    GWT.setUncaughtExceptionHandler(new
      GWT.UncaughtExceptionHandler() {
      public void onUncaughtException(Throwable e) {
        // do exception handling stuff
      }
    });
    DeferredCommand.addCommand(new Command() {
      public void execute() {
        onModuleLoad2();
      }
    });
  }

  private void onModuleLoad2() {
    // do module loading stuff
  }
}

2. Unravel GWT's UmbrellaException

Once your UncaughtExceptionHandler is set, you may notice that some of the exceptions caught are instances of GWT's UmbrellaException, rather than the actual exception thrown within your code. GWT uses UmbrellaExceptions to wrap exceptions in certain cases (loops, etc.), and it nests the root exception within it. For the purpose of easier debugging and more accurate error handling, it's often helpful to unwrap these exceptions, like this:

GWT.setUncaughtExceptionHandler(new
  GWT.UncaughtExceptionHandler() {
  public void onUncaughtException(Throwable e) {
    Throwable unwrapped = unwrap(e);
    // do exception handling stuff
  }  

  public Throwable unwrap(Throwable e) {
    if(e instanceof UmbrellaException) {
      UmbrellaException ue = (UmbrellaException) e;
      if(ue.getCauses().size() == 1) {
        return unwrap(ue.getCauses().iterator().next());
      }
    }
    return e;
  }
}

3. Log exceptions on the client

Now that you have a handler catching your rogue client side exceptions and also a handy method to unravel them to their root causes, you'll want to actually do something useful with these exceptions to help developers more easily diagnose and resolve the underlying problems. While the appropriate action may differ per application (e.g. present an error dialog, email the support team, etc.), at the very least the exception should be logged. Fortunately, GWT delivers a simple but powerful solution for client-side logging, piggy-backing off of the standard java.util.logging package. Plugging a logger into the UncaughtExceptionHandler is easily accomplished:

public class Example implements EntryPoint {

  private Logger logger = Logger.getLogger("");

  public void onModuleLoad() {
   GWT.setUncaughtExceptionHandler(new
     GWT.UncaughtExceptionHandler() {
     public void onUncaughtException(Throwable e) {
        logger.log(Level.ERROR, "Ex caught!", e);
    }
    // do module loading stuff
  }
}

Leveraging a helpful set of handlers that GWT provides, loggers can write to a number of different channels: development mode window, console, in-window pop-up, Firebug, and, the coup-de-gras, to the server(which I'll cover in a bit). These channels are easily configurable via your .gwt.xml file:

Logging in GWT is very well documented, so there's no need to retread here.

4. Log exceptions on the server

The majority of the logging handlers write to the client side, however in a production environment this often isn't much help to you. When your users stumble upon an error, it's unlikely (and actually unsafe from a security perspective!) to expect them to look in client-side log files to help you diagnose the problem. Instead, your application should do this work for you, by logging all unexpected exceptions to the server. Fortunately, remote logging of this sort is as simple as enabling it in the .gwt.xml config file:

...and then defining the remote logging servlet in your web.xml file:

  
    remoteLogging
    com.google.gwt.logging.server.RemoteLoggingServiceImpl
  
  
    remoteLogging
    /your-gwt-module-name/remote_logging
  

Now, with one simple call to the log() method in your UncaughtExceptionHandler, the exception is sent automatically to your server log. Note, however, that by default it's assumed that you're using java.util.logging, which may not necessarily be the case. To hook these logs into, say, Log4J, check out SL4J.

Finally, it's possible that instead of using GWT's remote logging, you may want to implement a simple custom RPC to log (and possibly do other things - e.g. send email, etc.) when an unexpected exception is encountered. For example:

GWT.setUncaughtExceptionHandler(new
  GWT.UncaughtExceptionHandler() {
  public void onUncaughtException(Throwable e) {
    myService.handleException(e);
  }
}

Seems straight-forward, right? Be careful! First, not all exceptions are serializable or follow the rules of serializable (ugh!). For example, GWT's JavascriptException inherits Serializable, but doesn't implement a public no-arg constructor, so when you try to send this over the wire via your RPC, a run-time Serialization exception is thrown. And since this exception would be thrown inside your exception handler, it would effectively swallowed. Second, the stack trace within an exception is transient, and so is lost from client to server (so if you need it on the server side, send it as a separate parameter). Which leads me to the next tip...

5. Consider logging exception stack traces

Of course knowing that an exception occurred isn't usually helpful unless you know exactly where it was thrown from! On the server side we take this for granted, since stack traces are typically logged by default with our exceptions. On the client side with GWT side, however, it's not so easy. When Java code gets compiled into Javascript, it is both shrunk and obfuscated. This is obviously useful for performance and security, but from the perspective of supporting applications, it makes things much trickier. For example, instead of seeing a stack trace with method/class names and line numbers, you might get something like this:

 Unknown.Jq(Unknown Source)
 Unknown.Tm(Unknown Source)
 Unknown.Sm(Unknown Source)

Not too helpful! Fortunately, there are different options for how GWT compiles into Javascript which provide different amounts of information for debugging. Of course nothing in life is free; the trade-offs are security and performance. If you wanted nice clean Javascript that resembles your Java code and that produces nice clean stack traces, then you can flip the GWT compiler "style" flag to either "pretty" or "detailed", however this would obviously increase the size the Javascript (and thus take longer to download)...and any user could more easily grock your client side code. There's a nice compromise though, which is to provide file names and/or line numbers in your stack traces, but to keep the source code obfuscated. This can be configured in your .gwt.xml file:

Note, again, that this does increase the size of your Javascript bundle. For one application I worked on, I found that just turning on the emulatedStack increased the Javascript by 30%, and turning on emulatedStack plus line numbers increased it by 100% (i.e. doubled the size!). Depending on the performance requirements of your application, this may or may not be acceptable.

Finally, it's helpful to know that even with the obfuscated stack traces, there is still hope for hunting down mysterious exceptions. What seems like jibberish is just obfuscated code, and you can manually translate the obfuscated method names (e.g. "Jq", "Tm", or "Sm" from the above example) to the Java method names using the symbolMaps file that GWT generates. If only there were a way to do this programmatically...

6. Use the StackTraceDeobfuscator

GWT provides a utility class called the StackTraceDeobfuscator which (as it advertises!) deobfuscates stack traces, translating the condensed, jibberish stack trace (like above) into a one that you would expect. If you're using GWT's remote logger, then it helpfully uses this by default, with one caveat: it knows where to find the symbolMaps file. GWT places this in the "WEB-INF/deploy/your-app/symbolMaps" directory, and you can instruct the RemoteLoggerService to look there accordingly:

  symbolMaps
  your-app/symbolMaps

You could also call the StackTraceDeobfuscator manually, if, for example, you had your own RPC service that handled exceptions in some custom way. I wrote a simple subclass of the StackTraceDeobfuscator, and called it like this:

public class LogServiceImpl extends LogService {
  public void logError(Throwable t, String stackTrace) {
    StackTraceElement[] sts = stackTraceDeobfuscator.deobfuscateStackTrace(t.getStackTrace());
    t.setStackTrace(sts);

    LOG.logError("An error happened", t);
  }
}

7. Throw exceptions explicitly in RPC Services

Now, to the server side (which, thankfully, is much less tricky!). Imagine you have a simple RPC service that throws a custom exception, MyRuntimeException, which extends from RuntimeException. For example:

public class MyServiceImpl extends MyService {
  public void foo() {
    throw new MyRuntimeException();
  }
}

By Java's rules, there is no obligation that you explicitly declare (either in the impl or interface) that foo throws MyRuntimeException. For the sake of simplicity, you may choose not to, however you might be surprised then to find that when you catch this exception in your AsyncCallback...

myService.foo(new AsyncCallback() {
  public void onSuccess(void v) {
  }

  public void onFailure(Throwable t) {
    if(t instanceof MyRuntimeException) {
      // do something
    }
  }
}

...the object t will not be an instance of MyRuntimeException, but rather an UnexpectedException. This is because GWT did not know to compile MyRuntimeException into Javascript, and so the client code does not know of this type. To remedy, make sure you explicitly throw MyRuntimeException in your service interface (but not in your Async interface!):

public interface MyService {
  void foo() throws MyRuntimeException;
}

Note that if you threw some subclass of MyRuntimeException, the instanceof check would work, however be aware that GWT would then compile all concrete subclasses of MyRuntimeException, which could slow build time.

Whew...that's it. Please share any tips you have, or let me know if you see anything awry with what I've written. Thanks!

The idea that the brain can do two things simultaneously has been myth busted. Instead, we’ve learned that we have just a serial processor, and although it may be deft enough to provide the illusion of multitasking, on the inside it’s just an executive controller quickly switching back and forth from one task to another.

But is there anything wrong with an illusion, if the result is the same?

Well, yes. The problem is that there’s a cost to this context switching. A big one. For starters, studies have shown that we are actually less productive overall while multitasking, and make significantly more mistakes. In addition, multitasking also inhibits learning, lowers IQ, and increases stress. And as though that wasn’t enough, it turns out the more we multitask, the worse we are at it. Whew.

Despite all we know about the problems and perils of multi-tasking, however, the possibility of unitasking in the modern software development organization looks bleaker than ever. Cramped or open office environments increase the possibility for disruptions from our physical surroundings, and the ubiquity of social networking technology (e.g. email, text, instant message, Facebook, etc.) invite interruptions from beyond.

What’s more, within the world of software development, there’s a strong trend toward agile techniques, which value above almost all else communication and continuous feedback. More interruptions!

Now it might seem at this point like I’m a luddite, a social recluse, or (gasp!) a waterfall advocate, but in fact it’s just the opposite. I’ve seen first hand the benefits of both Agile processes and new tools for communication, it’s just that I think (in some environments more than others) we might take it too far. Responsiveness is important, but as the science of multitasking shows, there’s such a thing as over-communication. For the sake of our productivity, our sanity, and the quality of what we’re building, at some point we need to tune out the world, sink into a state of concentration, and get some freaking work done. But how?

Pomodoro

The time management practice of Pomodoro seems to be getting more attention lately, especially within the Agile community. The idea is simple: set a clock for 25 minutes and allow yourself to work on one task, uninterrupted. At the end of the Pomodoro, you get a quick break, and then you start again. If something pops up during the Pomodoro (an email from a colleague, a phone call from the product owner, etc.) you’re advised to record it and get back to it when the 25 minutes are up. Sounds great, right?

In my experience, it’s not so easy. If a business owner pays you a visit from 2 floors up about a bug in the application, he may not be happy when you ask him to wait 17 minutes until you’re done with your “Pomodoro”. Basically, the world doesn’t hinge on your stop watch.

Even if it did, without knowing when you start or stop the clock, it’s impossible for a colleague to know when it’s appropriate to interrupt and when it’s not. Depending on the size/activity of a team, a half dozen interruptions (of different forms) are easily possible within a 25 minute span. Just acknowledging these is enough to lose concentration.

Lastly, if each developer is on his own Pomodoro cycle, you could easily (albeit absurdly!) reach a deadlock. Joe pings Bill during Bill’s Pomodoro, and so Bill defers till after, but when he does, Joe’s in the middle of his Pomodoro and defers as well, and so on. Eventually someone’s Pomodoro has got to budge.

Introducing Power Hour

What’s needed is a shared Pomodoro, whereby the team understands when the clock starts and stops, and collectively agrees not to interrupt each other during that time. This is exactly what my we’ve created on my current project, and we call it “Power Hour” (not to be confused with any bevarage-oriented games you may or may not have played in college).

So far it’s been simple and effective. Each day we have two Power Hours, one at 10am and one at 2pm. During those two hours we all go “heads-down”, and get work done. If one of us has a question for another during that time, we wait till after Power Hour to ask it. There are obvious exceptions (bugs in production, calls from directors, free food in the cafeteria, etc.), but for the most part the sanctity of the hour is respected (even, now, by others outside of the team).

It may seem that this technique might thwart communication, but in practice it just consolidates it, often to the 15 minutes before and after a Power Hour. If I need an answer to a question before I start my work, I know I’ll need to ask my colleague before Power Hour begins, and vice-versa. This is healthy trade – rather than a work-day full of randomly distributed interruptions, discussions organically converge into concise bursts.

Who knows if we’ll have the discipline to keep up our Power Hours, but for now they’ve been a big help in countering the pressures for multitasking. I’d be interested to hear your strategies for getting quality work done, efficiently. Please share!

(click to expand)

Step 1: Plan for the Test

Preparation is the trickiest part of usability testing. If good users are not found, relevant tasks not defined, or environments not set-up, the results of the test will be specious at best, or non-existent at worst! It is crucial, therefore, to develop the right foundation for test:

Find the Users: For the typical B2C web site, representative users can be found at any street corner or coffee shop. For many enterprise projects, however, users are not so easily located or accessed. Because users of enterprise software may have specific domain knowledge, work in different departments, or reside in different geographic offices, procuring access to them or aligning schedules can be a significant challenge. If possible, it is important to surmount these obstacles – real users will give more accurate feedback, and will be generally more receptive of the system when it is eventually rolled out, becoming advocates of it rather than opponents to it.

When it is not possible to find real users, representative users (e.g. mapping to defined personas), or even non-representative users, can be substituted with some success; many usability issues can be uncovered by users with no domain experience, a fresh set of eyes the most salient requirement.

It is well documented that usability testing requires no more than 4-6 users per round. Usability issues are typically universal and so are found early – there are diminishing returns for testing with any more than 6 users (i.e. they all tell you about the same problems!).

Plan the Tasks: Not every feature of an application can or should be usability tested. As noted above, it is important, therefore, to identify for usability testing only those features which are prominent, complex, or have a high cost of user error. Once these features are identified, a script of user tasks can be created. Note that tasks should clear, concise, and worded in a way that is independent of the design of the application (e.g. not “click ‘add item to cart’ and click the ‘purchase’ button” but rather “buy a widget”). Lastly, it is important both to run through the script yourself, and also ask another team member to do the same, and ensure that the script takes no more than 15-30 minutes to execute (with the assumption that a user unfamiliar with the application could take 3x this long).

Write the Post-Test Questionnaire: Depending on the application it may be useful to learn about not only the user’s performance (i.e. can they use the system) but also their preference (i.e. do they like it). Surprisingly, it has been found that the two do not always correlate – users may have problems using the application but say the like it, or use it fine but say they hate it. For this reason, it is important to supplement the observational data (i.e. notes from watching the user) with data from a post-test questionnaire. Note that demographic, qualitative, and quantitative data are important – so ask both open-ended questions like “how did you feel about XYZ?” and “on a scale of 1-5, was the system usable?”. Lastly, the late usability expert Randy Pausch advocated always asking at least one question after a usability test: “what were the two worst things about the system?”, guaranteeing that the user will be forced to shed light on at least two tangible usability flaws (and not try to avoid hurting your feelings).

Set up the Environment, Data, and Access: Finally, a stable and perhaps isolated version of the system must be available for the test participants to use. In many cases this will be the normal QA environment, but depending on the frequency of builds or transiency of the data, this may or may not be adequate. If it is not, alternatives may be either to deploy the system on your local machine or to create a special, temporary environment specifically for usability testing. The feasibility of these options obviously depends on the complexity of the application and dependent sub-systems, serivces, or data stores – a simple single web-app to single database may be easy to configure and deploy, whereas a complex system that leverages LDAP, multiple databases, an ERP, and a handful of external web-services would not be.

Additionally, for some usability test scripts, it will be necessary to have each user test with a fresh, pre-configured, non-confidential set of data (for example, maybe a user is asked to find the user with an outstanding balance, and so such a user must exist). If this is the case, a set of SQL set-up and tear-down scripts may be necessary, or perhaps just a script for manually re-creating the data via the user interface. In any case, it is imperative to understand the data dependencies for the test up-front, to ensure that tests go smoothly, and users find issues related to usability and not bad data conditions.

Lastly, in many cases the test subject will log in as a pre-configured, sample user, however in some cases it will be helpful for the user to log in as himself and see “his” data during the test. In this case, it is important to ensure the user has appropriate access control for the QA environment – and it’s not until he tries to log in that you discover that he doesn’t have authorization.

Step 2: Perform the Test

If planning is done correctly, actually performing the usability test should be a snap – your job is simply to make the subject feel comfortable, stand back, and watch what they do! Here are a few things to keep in mind.

Introduce Yourself and the Test: When meeting the user, it’s important to set a very informal, relaxed, and open atmosphere early, so that the user feels comfortable sharing how they feel about the system. To that end, be sure to mention the following simple points:

• It is the system that is being tested, not them.
• The system has issues, but it’s difficult for developers to see them because we are too close to the problem. This is why their help is so important.
• Negative feedback is actually more helpful than positive. No feelings will be hurt if they are frustrated or don’t like something.
• Their help is greatly appreciated, and will contribute to a better product.
• If they get stuck you can jump in and help, but otherwise you’ll stay quiet and let them drive.

After introductions, give them the test script and have them briefly skim the tasks. Convey to them that although the tasks should only take 30-45 minutes, they should feel comfortable taking their time.

Observe, Ask Questions, but Don’t Help: Once the user has begun, it’s important to let them use the system as they wish, even if it means them navigating down false paths (the whole point is to see how they would use it if you weren’t there!). As they complete the tasks, encourage them to talk, and be sure to ask probing, but non-revealing, questions, like “I saw that you clicked the ‘filter’ button but were surprised with the result – what did you expect would happen there?”. If the user is hopelessly stuck, then it may be necessary to give them hints, but otherwise remain an observer.

Take Notes: As the user completes the tasks, take copious notes of their actions, comments, and frustrations – these notes will trigger your memory later and fuel the Usability Observations list that you assemble. Many would argue that it is necessary to also video record usability test sessions, however, the benefit is rarely worth the cost in my opionion (even though it is relatively cheap): project stakeholders seldom take the time to actually view a usability test video session, users may be apprehensive and less honest when being recorded, and the set-up and storage of the recordings can be frustrating and time-consuming.

Step 3: Analyze the Results

Once the tests have been completed, the data must be analyzed and then synthesized into a set of actionable recommendations. To do so, two artifacts can be created, an objective Usability Observations List and a subjective Usability Analysis & Recommendation document.

Create a Usability Observations list: Go back through your notes, and record discrete usability issues that users encountered in a simple spreadsheet. The list should be objectively and tactfully worded, stating, for example, that the user “did not know what the word ‘invoice’ meant” and not “‘invoices’ should be called ‘bills’”. Consensus should be noted for each item (e.g. “3 out 6 users”), and each should be classified either as a “usability bug” (which would later be entered in an issue tracking system to be implemented during the current sprint) or as a “feature request” (which would be added to the product backlog, to be prioritized for a future sprint).

Create a Usability Analysis & Recommendation document: Finally, it will be helpful for project stakeholders to receive a high-level analysis of the usability test, summing up overall impression of the site’s usability (using both quantitative and qualitative data as useful), the areas of the system that were most and least usable, and a set of recommendations for future action (e.g. new features to the backlog, additional rounds of usability testing, etc.). This document may captured in a Wiki, stored in document control (e.g. SharePoint), or just sent via email.

That’s it! Again, I’d like to hear your thoughts on usability testing on enterprise projects. Please share!

On most enterprise applications, in my experience, usability is promoted via a number of low-cost methods: following usability standards, deferring to user interface specialists, trusting our intuitions, or simply conferring with customers and business owners. And while these methods are sufficient in most cases, when usability is a high priority, there is simply no substitute for actually watching people, without interference, use the software we build. This is the practice of usability testing.

In particular, I believe that usability testing should be strongly considered on projects where…

1. The user base of the system is numerous, not technically-savvy, or external to the customer.
2. The cost of user error is high (e.g. software for managing financial transactions, medical procedures, etc.) and features are complex.
3. The UI framework or paradigm is nascent or highly flexible (e.g. RIA, iPhone, etc.) and a consensus on usability patterns or best practices has not yet been reached.

In addition to making our products easier to use, usability testing can also be of great benefit to us, the developers of the software, by improving our usability intuitions, educating us on the functional domain and application of the software we build, and re-enforcing for us that what we create has a tangible impact on real people.

In this post I’ll describe usability testing within agile projects from a project management perspective, describing how to estimate, prioritize, and “burn-down” the usability test. In a follow-on post, I’ll dig into the actual activity of usability testing, breaking down some best practices within the framework of a simple 3-step process: (1) plan for the test, (2) perform the test, and (3) analyze the results. Let’s get started…

Managing Usability Testing with Scrum

Effective usability testing can be easy, quick, cheap, and, as the diagram below shows, can fit seamlessly within the Scrum methodology. Here are some tips, given my experiences with usability testing on Agile projects, on how this can be achieved:

(click to expand)

Usability Test as Backlog Item: Unlike other development best practices (like code reviews or unit testing), usability testing should, in my opinion, not be “built-in” as a part of normal development tasks, but rather should be treated as a separate backlog item - given an estimate by the development team, a priority by the business owners, and is assigned to a sprint for completion. Unlike other backlog user stories or tasks, however, usability testing has the property that it is never truly “done” - there are cases when (indeterminately) many rounds should be performed, and so the task of usability testing should never really be permanently plucked from the backlog.

Estimation: Although the amount of time it takes to perform a usability test depends on many project factors (e.g. scope or complexity of the application, availability of users, etc.), in my experience (using the methodology to be laid out in the next post) effective usability testing can usually be completed with roughly 30 man-hours of work: 15 for the preparation, 10 for the test itself, and 5 for the analysis and communication of results.

Prioritization: Oftentimes, even when usability testing should be performed (see above), business owners or other project stakeholders will still be reluctant or apprehensive to do so - especially when other methods (like just trusting intuitions) are cheaper and seem just as effective. It is important in these cases to stress that (a) usability is important, (b) we are notoriously bad judges of what is usable (too technical, too close to the system, etc.) and (c) usability testing can be cheap and extremely fruitful. Further, it is crucial to note that the earlier in the lifecycle that usability issues are identified, the easier they are to resolve - for example, redesigning a system’s confusing navigation structure once it is in production can be extremely costly and risky. In the end, however, usability testing should be prioritized along side all other user stories on the backlog, and this prioritization should be driven by the product or business owner.

Usability Testing as a Burndown Task: Once usability testing has been selected for inclusion in a sprint, it should be assigned to one developer (where in agile teams, “developer” can mean tester, programmer, UI designer, etc.). Note that all activities of usability testing (planning, testing, and analysis) should fit within one sprint, and the task should be “burned-down” like any other development task. The practice of usability testing does not require an “expert”. Anyone is capable of performing one, and in fact, should if given the opportunity - it can be a rewarding and broadening experience.

Incorporating Feedback: Once the usability test is performed, observations from the test will be analyzed and discussed (see next post), and eventually some consensus will be achieved regarding how the application should change to improve user experience or efficiency. This feedback can flow back through the Scrum development process in one of two forms: either as (1) new user stories added to the backlog (e.g. “as a customer, I want to be able to use wild cards in text searches”) or, when improvements are simple, as (2) bugs managed by the defect tracking system (e.g. “error messages should be red, not yellow”).

As always, I’d be happy to hear about your experiences with usability testing on Agile projects. How often do you usability test? How do you track it from a project management perspective? Please share!

Also, check back in a few weeks for part 2 where I’ll dig in to the actual practice of usability testing in more detail.

Imagine that some for some feature you're working, it could be implemented in a number of different ways, each with some impact to maintainability, performance, reusability, and development effort. You probably have a strong sense for which path is best, having looked at the problem in detail, but it's not usually wise to do anything until others (e.g. managers, business owners, other developers, etc.) weigh in. Until the decision is made, however, your task is blocked...and the sprint deadline looms.

In these cases, your job is to pop this decision up to the right group of people and facilitate the best decision, quickly so you can get back to work implementing the feature. Pretty simple, right? Just schedule a meeting, present the evidence, make your cogent argument, and they're bound to see the light. Err...not always.

In my experiences, facilitating a big decision is one of the more precarious, but crucial, tasks of the enterprise developer. Even if you get the right people in the room to make the decision, the discussion is like walking a tight-rope - one wrong step and you fall into the pit of indecision, or worse the abyss of irrelevant pontification.

So what can we do, as developers, to facilitate good, collective decisions? One useful thing to consider a psychological phenomenon called decision fatigue, defined as:

"the deteriorating quality of decisions made by an individual, after a long session of decision making" (wikipedia)

On large enterprise projects, decision fatigue is (depressingly!) the standard mode of operating for many of us. On any given day, we all must navigate and make decisions on dozens of complex, confusing, and frustrating issues, and this all takes mental energy. The more choices we make, the more tired our brains get, and according to theory of decision fatigue, the more likely that we'll then make bad decisions, or defer the decision altogether (a decision in itself!)

Back to the original example then, how can you help the group not fall victim to decision fatigue, so they can come to a good decision and you can get back to your task? Here are a few strategies:

1. It's always best to schedule morning meetings for important decisions. In a recent study on parole hearings in Israeli prisons, it was found that judges gave out parole 70 percent of the time when appeals were heard in the morning, and only 10 percent of the time when they were heard in the afternoon. Essentially, after hours of listening to arguments, decision fatigue sets in, and judges would rather make no decision than the wrong decision. Anecdotelly, this seems right - how many afternoon meetings end in the insanely-frustrating: "let's schedule another meeting!" - i.e. decision deferred.

2. If decision fatigue is the result of making hard choices, then another answer is to make choices easier for the group. One way to do this is to limit the solution set. For example, a study discussed in the book The Paradox of Choice found that consumers were 10 times more likely to buy jam when they were presented with 6 flavors rather than 24 - in other words, too many options make decisions difficult. As developers, when presenting decisions to the team, it's important therefore to frame the decision in the most "cognitively fluent" way possible, identifying ahead of time the salient trade-offs and simplifying to a manageable set of possible solutions.

3. Lastly, if the first two tips don't work, bring some candy for the team. Seriously. In studies done by Roy F. Baumeister, it was shown that sugar (specifically glucose) can effectively rejuvenate the brain from its decision fatigue, improving self control and the quality of decisions. (Note that it's this effect that is partially responsible for the junk food you buy when grocery shopping on an empty stomach - your brain's out of fuel, and so it makes poor decisions.)

Obviously, there's plenty more to consider when helping a group come to a big decision, but being aware of decision fatigue might help. As always, let me know what you think! (and if you're interested in some of the research, here's a great starting point.)

You and three friends stumble upon a great idea for a tech start-up. Because you're all talented developers and there are no "business" people to constrain you, the only thing that separates you from monumental success is just building the darn application. (seriously, it's a great idea). So you decide to all sit in a room to brainstorm the initial architecture, then you'll divvy up the pieces and get started.

I've lived this scenario a handful times, and have seen it too many to count.

What you'd expect to happen is this: each developer will come to the table with a set of preferences on what the architecture should look like, based on their unique knowledge and experience. All of these individual preferences would then be melded together, and the resulting architecture would represent some sensible compromise or average of the pre-discussion opinions.

What actually happens, however, is the exact opposite. Instead of ideas being pushed toward some average, they get pushed to the extreme. Cutting edge technologies, patterns, and approaches are chosen over relatively stable ones, and before you know it, you're programming the system in Clojure using a NoSQL database built using Gant, none of which anyone has any experience with. Two months into development the team realizes that it spent most of its time learning the new technologies, and very little has been actually produced. At this point the "window" has been missed, and everyone goes back to their old lives.

Some of what's going on here is the law of group polarization:

"Members of a deliberationg group predictably move toward a more extreme point in the direction indicated by the members' predeliberation tendencies" (Sunstein)

This phenomenon has been consistently replicated in a number of studies: conservatives meeting with other conservatives form more conservative beliefs, liberals more liberal, racists more racist, and so on. It's why political parties hold conventions before elections, and why niche discussion boards on the internet end up going completely off the rails. Like-minded peoples' preferences push toward the extreme.

The cause of this phenomenon seem to be two-fold:

First, people want to be perceived favorably by others, and so once they hear what others think, they adjust their opinion accordingly, to fit in with the group. In terms of the original example, who wants to be the guy who suggests to write the new application in old, boring Java? Come on man!

Second, people will share evidence in support of their opinions, which will further influence others in that direction (in which they are already leaning anyway). Essentially, the argument pool will be limited to evidence on only one side, and so this preponderance of evidence will push everyone even further.

Back to software development, group polarization happens often, potentially any time some relatively homogeneous faction (e.g. developers, managers, business owners, etc.) meets in isolation and makes decisions without input from others. The antidote, as would be expected, is inviting diversity...which is one reason that Agile works so well, since it puts developers in the same room with business owners. Although there may be disagreements, the resulting decision is typically some compromise.

Anyway, I find it helpful to be on the lookout for group polarization. And if I ever do a start-up again, I may be that lame guy who suggests Java!

1. Don't check in code right before you go home

It's 6:00, the [wife|kid|girlfriend|dog] is expecting you home in 30 minutes, and you just finished a really simple change. There's no harm in just checking this code in now before you go, right? DON'T DO IT! I swear I have been burnt by this too many times to count. If the build breaks or some bug pops up, you won't be there to resolve it...and every minute it goes unresolved you'll be losing good will with your fellow developers and testers. Obviously I always run a local build before checking in, but now I go a step further and try to hold off on any check-in if it's within an hour of me going home - better safe than sorry!

2. Batch your questions

As a new developer (or developer new to a team), you'll have loads of questions. This is good. Don't keep these to yourself, however don't ask them one-by-one as they pop into your brain. Write them down on a list, and only when you can go no further, then ask the tech lead/analyst/business owner all of your questions in one batch. People are almost always happy to help, but if you can limit the interruptions, it's always appreciated.

3. Stop complaining!

Yes, the system you work on is insanely complex/convoluted, the business owners are totally irrational, and the development team dysfunctional. Still...don't complain. If enterprise software development was easy, businesses could (and would!) hire people a lot less smart than you, and pay a lot less money. The fact that things are frustrating just means that talented developers are needed, and that we make a good living. If things are so bad, then quit - there's always a market for good developers. Otherwise, do what you were paid to do: make things better!

4. Keep it Simple - Format Matters

We all know the virtues of writing clean, readable code, but we don't always apply this rule to the artifacts we produce: emails, documents, models, etc. Spending time tuning/formatting these other products not only makes our ideas easily digestable by others, but it makes our ideas more persuasive as well. The minute it takes to tune an email for the people reading it is well worth it.

5. Be Nice!

I'm not talking about "nice" in the traditional sense, but rather in a game-theoretic sense. Many development activities can be boiled down to simple prisoner's dilemma situations: e.g. if we all comment our code we're better off, but individually it's more rational to be a free-rider and just take advantage of everyone else's comments (which is why it's sometimes irrational to comment code. It's been found that many people naturally play a tit-for-tat strategy, meaning if everyone else is "nice", they'll be nice too, but when someone free rides, they'll follow suit. Given this, it's always good to play the nice strategy and help others (review code, unit test, etc.). As soon as you don't, others won't either, and your development team can quickly turn to an every-man-for-themselves.

Now there are some strong reasons to model these entities in separate tables in your relational database, and there is generally not much debate here. Where things get a little tricky is when this reference data propagates up into the application tier. In my experience, developers employ a number of different strategies to solve this problem, and so in this post I'll define some of the more popular approaches and describe the pros and cons of each. As always, I'd love to hear your thoughts, so please share. Here goes...

Types of Reference Data

Let's first start with an example. Imagine your system has customers, and every customer has some loyalty level (e.g. "Gold", "Silver", etc."). The tables could be designed simply as follows:

 

 

...and the data in the LOYALTY_LEVEL table might be...

 

 

This is a pretty vanilla example, so it would be tempting to jump right in and translate this into the application tier, but before we do it's important to stop and ask a few questions first:

1. Does the reference data need to be retrieved independent of the master data (e.g. display all loyalty levels in a drop-down, etc.)?
2. Are there other attributes beyond just the normal id-name pair? For example, "description", "abbreviation", or something with more business meaning like "percentage discount", etc.?
3. Do specific values need to be hard-coded into the business logic? For example, could there be logic if (cust.loyaltyLevel == "Gold") { sendOvernightMail() } ...or something similar?
4. Is the reference data at all dynamic? For example, could there be a new loyalty level at some point (e.g. "Diamond" or maybe "Coal"!)? Further, is this reference data very dynamic in that it could be added by a user via some UI, or would a new value be a configuration change?

Now I can hear the objections already: "hey, not all of these types of 'reference data' are really reference data!". And that's actually the point. Oftentimes, in my experience, we fail to appreciate these four distinctions, and we shoe-horn all entities that even roughly resemble reference data into the same solution, to sometimes pernicious consequences. For example, we may treat States the same as Loyalty Levels in the application tier, even though the former is probably static, not used in business logic, and will only ever have two-attributes, while the latter could be the exact opposite.

To best understand how to design for different types of reference data, it's helpful to lay out a few common solutions...

1. Flatten in Model Class

An unsophisticated (but not uncommon) approach is to flatten the reference data in the containing model object. For example, the customer class could look like this:

 public class Customer {
    private String name;
    private String loyaltyCode;
    private String loyaltyName;  // optional
  }
  

Obviously this will work, until a new attribute needs to be added to a Loyalty Level (e.g. "description")...or Loyalty Codes need to be retrieved independent of Customers (e.g. to populate some drop-down), in which case you'll probably want to define a...

2. Custom Class

A custom LoyaltyLevel class could be created with the two attributes "code" and "name"...

  public class LoyaltyLevel {
    private String code;
    private String name;
  }

...and then composed within the Customer class:

  public class Customer {
    private String name;
    private LoyaltyLevel level;
  }

On the up side, the LoyaltyLevel class could be fetched independent of Customers (e.g. to populate a drop-down, etc.), and attributes could easily be added to the LoyaltyLevel class without adding unnecessary bloat to the Customer class.

The problem comes when you want to create some business logic that refers to some specific loyalty level. Using the example above, it's possible that some business rule should be written such that customers at the "Gold" level are treated in special manner.

  if("G".equals(customer.getLoyaltyLevel().getCode())) {
    sendOvernightMail();
  }
  else {
    sendViaHorseAndBuggy();
  }

Again, this works, but there are a few problems. First, the "code" attribute in LoyaltyLevel is not type-safe, since it's just a String in the LoyaltyLevel class, and so there's not guarantee that an invalid value (e.g. "X") couldn't creep in. Second, we're hard-coding the level, "G", into our business logic, and while this could be ameliorated with a simple String constant, again, this is not type-safe (and a general anti-pattern). And finally, it's very easy to forget to write bugs where "G" is compared with the LoyaltyLevel object itself and not it's String code. For example:

  if("G".equals(customer.getLoyaltyLevel()) { // forgot .getCode()!
    sendOvernightMail()
  }

In other words, if specific LoyaltyLevels are to be used in the business logic, it's much preferable to keep them typesafe in an...

3. Enum

A LoyaltyLevel enum could be created in the application tier:

  public enum LoyaltyLevel {
    Gold("G", "Gold"),
    Silver("S", "Silver"),
    Bronze("B", "Bronze");

    ...
  }

...and a type handler construct in your ORM tool (like Hibernate's or IBatis, etc.) can be used to easily load this into the Customer object on retrieval from the database:

  public class Customer {
    private String name;
    private LoyaltyLevel level;
  }

Voila! This solves our problems, right? LoyaltyLevels can be retrieved independent of Customers, new attributes can be easily added without affecting the master data, and specific instances can be easily (and type-safely) referred to in business logic (e.g. "LoyaltyLevel.Gold == cust.getLoyaltyLevel()").

Not so fast! What happens when the business wants to add a "Diamond" level? Or wants to get rid of their "Bronze" level and replace it with an "Aluminum"? In either case, both the enum and the database need to change (since both redundantly define the Loyalty Levels), which obviously necessitates a code/configuration change (i.e. not something that could happen via some UI). In cases where the reference data is very static, this may be fine, but if Loyalty Levels are to be added with any fluidity, encapsulating the different values in Enums won't work.

Analysis and Recommendations

So I've presented 3 solutions, and as the table below shows, each solution has its drawbacks. Things are less clear now than before! What gives?

 

 

In the end, is there an good answer? Well, I've found the following two heuristics to be helpful:

1. If the reference data is static, has few attributes, and is used in business logic, use an Enum. The redundancy of reference data defined both in both an enum and a table in the database is relatively harmless (since the data is static!), and worth the cost for gaining the benefit of type safety, clarity, and ease of use (in my opinion).

2. In all other cases, use a Custom Class. In cases where the business logic seems to reference a specific value, rather than hard-code (e.g. "G".equals(code)), try to refactor this such that the data makes the decision rather than the code. For example, in the example of Loyalty Levels and shipping methods, a new column added to the Loyalty Level table...

...and this would effectively eliminate the dependency on a specific reference data value in the code and move it to the database.

3. If you absolutely need to hard-code the "code" values of the reference data for business logic, then do so with constants in one place and as close to the reference data class as possible (preferably in it). Do not sprinkle hard-coded values (e.g. "G", "S", etc.) around your business logic. (I know...duh!)

Anyway, I'd love to hear your thoughts. Am I missing something? How do you manage reference data in the application tier?

The question is, then, how do you get started? What do you do on that first day? You've committed to completing all the tasks, so does it matter? Of course it does.

Two simple approaches are...

1. Start with the tasks that are most fun. "Investigating clustering" sounds interesting, so start with that! This makes for a more enjoyable sprint...at least for the first few days.

2. Knock off the low hanging fruit first, and get some quick gratification at the beginning of the Sprint. Just like some financial gurus say to pay off your lowest debts first, it's nice to build some confidence before you get to the tough stuff.

Honestly, I find it really tough not to use these strategies - I just naturally want to do the fun tasks or quick-wins first...but having lived through more than a few failed sprints, I've quickly learned better.

First, there are some tasks which other developers are depending on me for (e.g. "Upgrading GWT"), and so if I wait till the last few days to do this, I effectively hose my colleagues. Second, there are times when I don't finish all of my tasks (gasp!) - maybe I had to call in sick one day, or requirements shifted, etc. If I do the most fun tasks first, I may have left some high value tasks hanging.

So here are some more disciplined approaches:

3. Start with the task that has the most dependencies on it (either within my own task list or for other developers). This approach is the best for keeping you in the good graces of your team.

4. Find out from the business owners which tasks are the highest value and work from high value to low - that way if something doesn't get done, it will be less of a big deal. For example, the "Order Status" screen might be crucial to the business, but "Upgrading to GWT" is not as big a priority.

5. Dig into the task that is the most complex first, so you can identify and mitigate the biggest risks right away, and you'll have more time to address them during the sprint than if you waited to the end.

Now each of the last three approaches (which can be blended, of course), are significantly more disciplined than the first two, but there's still a problem...

Most developers (me included) typically like to work sequentially - complete a task, move on to the next one. I find this more gratifying (and less stressful), but not very effective. It's very possible that lingering within each of my tasks is some big gotcha, that needs time (i.e. calendar time, not hours of work) to be dealt with. For example, maybe there's some question about the "Order Status" screen that needs to be posed to a business owner, and that business owner is booked solid till Thursday. Or maybe investigating clustering requires the assistance of a sys admin, and he needs a week lead time. If I don't identify these dependencies early, I could easily put myself in a position where I can't complete my tasks.

Given this, I've found the most effective approach to starting the sprint is this...

6. On the first 2 days, take a spike through each of my tasks (similar to the XP concept), understand better the requirements, and pick out the tricky pieces that might require input from others. This may require writing some code, but not much. Once I have a good handle, at a conceptual level, of what each task entails, then I can use some blend of strategies 3, 4, and 5.

The biggest drawback to this approach is that on those first two days, I find myself barely burning any work down - because most of what I'm doing is asking questions and planning. After everything is in order though, I typically can roll smoothly through my work.

Anyway, what are your thoughts? How do you start your sprint?

For example, in any given hour, I could easily receive one more email from project manager, phone call from business owner, chat request from developer, one-on-one drop-in from architect, build failure notification, and bug submission. And every single one of these would result in the same thing: more work on my to do list. Now add these to my existing tasks for this sprint, and all I can say is "holy crap".

I'm sure it's no different for anyone else doing the enterprise development thing, so my question then is this: how the heck do you manage this all? Or more specifically, how do you remember all that you need to do? And at any given point in time, how do you know you're doing the highest priority thing?

Thinking on this a bit, I know there's probably not much I can do about the quantity or frequency of incoming to-dos - working in a team environment is just inherently chaotic and complex.

My main frustration then is really just that there are so many lists of to-dos for developers: bug tracking systems, project management tools, IDEs (e.g. TODO), post-it notes, etc. When I want to know what I have to do, I either have to consult each these multiple to-do "bins" or just try to keep it all in my brain (which just means I'm thrashing all day ).

Basically, what I really need is a single place to track all my development related to-dos (ala GTD).

For a long time, I tried to do this old-school: with a simple, paper notebook. Every time I identified something new to do, I'd write it down. And while this is great for visualizing the entire list...and there's nothing better than physically crossing something off...there were obvious short-comings: I didn't always have my notebook on me, I was constantly writing and re-copying lists, there was no way to search or prioritize items, and there was no notification feature (that I was aware of ;).

More recently, I moved everything to my smart phone into a program called ToDo Matrix (good, but kind of over- priced). This works better than paper (more structure, searchable, etc.), and I've been able to consolidate most of my to-dos into one master list, but I still have a problem with redundancy (i.e. some to-dos are in the Bug Tracker, some in Rally, etc....and I either have to make a duplicate to-do on my phone, or keep them where they are and just consult multiple lists).

So does anyone out there have a good strategy for ordering the chaos of a mere mortal enterprise software developer? How do you manage your to-dos?

Even still, implementing code splitting can be a bit tricky, especially deciding where in your framework to build it in. There are some helpful constructs and patterns out there (e.g. AsyncProxy, Async Provider, etc.), but it’s still not really clear if there is a definitive best practice for code splitting in large applications using GWT MVP. So…for the purpose of (hopefully!) sparking some conversation on this topic, here’s a solution we came up with on a recent project. Please feel free to share your solutions…or any thoughts/criticism you have. Here goes…

Scenario – Application with multiple “Sections”

Imagine an application that has multiple distinct “sections”, where each section contains lots of “things” (places, dialogs, widgets, etc.), and further that a “thing” can only live in one section. For example, consider a simple app with three sections “A”, “B”, and “C”.

Again, anything that is one section cannot be programmatically referenced by something in another section. In other words, the package dependencies should look like this:

If an application can be segmented in this way, then the “section” could be a good boundary line for code splitting – in other words the application could be chunked in four pieces: main, A, B, and C. When the user first loads the application, only the main chunk is downloaded. When he navigates to any place within section A, for example, then all of section A is downloaded, and similarly for sections B and C.

Now sections may not always be good split points (perhaps they are too big, or there are inherent, un-refactorable dependencies from one section to another), but if they are, here’s how it can work…

Place Subclass

Create a subclass of the GWT Place, from which all places in the application are extended. This base class has a reference to a custom enum “Section”, which defines the different sections in the application (e.g. “A”, “B”, “C”). In psuedo-code it looks like this (though note there’s a lot missing from the MyPlace class):

 
public enum Section { 
    A, B, C;
}

public class MyPlace extends Place { 
    private Section section;
    ...
}

Again, since all places have a section, the URL for a specific place might look something like “#someplace:section=A”.

Composite and Section ActivityMappers

Now each section will have it’s own ActivityMapper (e.g. AActivityMapper, BActivityMapper, etc.) which instantiates the individual Activities within its section. Splitting up into separate ActivityMappers like this ensures that there is no one master ActivityMapper that has dependencies to all Activities in all sections (thereby breaking the dependency arrows, and thus code splitting).

These section-based ActivityMappers are glued together using a new, custom “composite” ActivityMapper, which, onPlaceChange, inspects the Section of the Place, and then delegates to the appropriate custom section-based ActivityMapper.

 
public class CompositeActivityMapper extends ActivityMapper { 
    private final HashMap mappers = 
               new HashMap();

    public Activity getActivity(Place place) { 
        MyPlace myPlace = (MyPlace) place;
        return mappers.get(myPlace.getSection()).getActivity(myPlace);
    }

    public void addActivityMapper(Section section, ActivityMapper activityMapper) { 
      mappers.put(section, activityMapper);
    }
}

Custom PlaceController

Finally, and this is the crux (!), the PlaceController is sub-classed, and loads the appropriate section’s ActivityMapper in the runAsync, thereby drawing the split-point boundary around the entire section. Essentially, the first time the user navigates to any place within a given section, the PlaceController will load that section (i.e. send down that section’s Javascript code bundle).

 
    public MyPlaceController extends PlaceController { 
   
        private CompositeActivityMapper compositeActivityMapper = new CompositeActivityMapper();

        private HashMap activityMappers = 
                  new HashMap();

        public MyPlaceController(CompositeActivityMapper compositeActivityMapper) { 
            this.compositeActivityMapper = compositeActivityMapper;
        }

        public void goTo(Place place) { 
            MyPlace myPlace = (MyPlace) place;

            switch (myPlace.getSection()) { 
                case A:
                    GWT.runAsync(new RunAsyncCallback() {
                        public void onSuccess() {
                            if(activityMappers.get(A) == null) { 
                                activityMappers.put(A, new AActivityMapper());
                                compositeActivityMapper.addActivityMapper(Section.A, activityMappers.get(A));
                            }
                            parentGoTo(myPlace);
                        }
                        public void onFailure(Throwable t) { 
                            // handle somehow
                        }
                    }
                    break;
                 case B:
                     ...
            }
        }

        public void parentGoTo(AdminPlace place) {
            super.goTo(place);
        }
    }

Conclusion and Caveats

Obviously, this is a high-level description, but hopefully you get the idea. Through the creation of a few custom infrastructure classes (e.g. Place, ActivityMapper, PlaceController, etc.), code splitting can be managed at a section level relatively seamlessly to the end developer – i.e. as long as the developer doesn’t break the dependency structure, it will work.

Finally, here are a few quick caveats:

• The compile report is an essential tool for understanding and managing dependencies for the purpose of effective code splitting. JDepend is also a useful tool for just the dependency portion.
• Obviously my post leaves some details (e.g. GIN, etc.) out, but they should be orthogonal.

Anyway, I’d love to hear other strategies for code splitting in MVP-based GWT apps – please share. Thanks!

Example: Web apps, GWT Modules, and EntryPoints

It's probably best to start with a simple example. Imagine you already have some ecommerce site (you can decide what it is that you sell!), and now you need to build an internal application to manage your business. There are two "sections" to this site: (1) the Inventory UI through which Customer Service Reps can see what's in stock, and (2) the Orders UI through which they can track orders and through which the Shipping Manager can fulfill orders and send them out.

Note that there is some code that both the inventory and orders sections will need to share ("common" above) - for example, base classes, domain classes, helpful utilities, etc.

The question is how do you structure this application within GWT? Here are a few choices:

1. Two WARs: have a separate web application for the inventory and orders sections, each sharing the common.jar file.
2. Two EntryPoint modules: have one web application, but create separate GWT modules for common, inventory, and orders, where inventory and orders both have EntryPoints defined (see picture below).
3. One EntryPoint module, three dependent modules: have one web application, but split each of the three pieces (common, inventory, and orders) into their own GWT modules, and then create one main GWT Module with an entry point which imports the others (see picture below).
4. One big GWT module: have one web application and just one GWT module and one entry point.

Just to be clear, here's a picture of the GWT module organization in options 2 and 3:

This decision is pretty fundamental, and so it's best to get it right up-front (or at least to try!). Here are some factors to consider:

Download Time

GWT applications are not like more traditional web applications. When the user first navigates to the site, he downloads the entire application (as a Javascript bundle), rather than downloading the HTML for each page incrementally as he navigates through. This means, all things being equal, the larger the application, the larger the Javascript bundle, and so the longer the initial load time of the application. The problem is, however, that if a user never accesses a whole section of the site (e.g. Shipping Managers never use Inventory), he still needs to wait for that portion of the Javascript to load. And if there are many unused sections in the site for a given user, this could mean a lot of wasted time.

At first blush, this may seem like a legitimate argument for either options 1 or 2 above (separate web applications or separate modules with entry points), especially if the different sections are sufficiently large. And while taking this approach and breaking things out could definitely solve your download latency issue, before you do this there are few caveats to first consider.

First, GWT 2.x has essentially solved this problem with code-splitting, a mechanism that allows you to partition your code, and only download portions as the user needs them, not all at once. (Note: structuring the dependencies to allow for effective code splitting can be tricky, so it is imperative that you think this through up-front). Second, Google is pretty clever in how it can boil down even a relatively big application into a small amount of Javascript, so unless the application is very large, this is usually not an issue. And lastly, the Javascript for the application is cached on the client, so only the first visit to the app could potentially be slow.

Looking at option 3, the different modules provide good possible division lines along which to code split, however code splitting is equally possible with option 4 (one GWT module) as well. Google has some helpful patterns defined for code splitting, and there are a number of other good sources as well (like here).

Essentially, application download time, in and of itself, is probably not a solid reason for breaking an application into multiple applications (either WAR or entry point), however if navigation between sections is rare, then it may be justified, which leads me to my next point...

Navigation and URL

With GWT, one application is just one URL (e.g. http://www.foo.com/index.html), and different "pages" are just inner links (e.g. "index.html#my-orders"). It may be important (for usability, SEO, security, etc.), however, for different "sections" of the site to have different URLs (e.g. "/orders/index.html" and "/inventory/index.html") rather than just one with inner links (e.g. "index.html#section=orders" and"index.html#section=inventory").

A most obvious solution would be to split it into different web applications (option 1), but in many cases (including the example above), this won't work - e.g. the two sections may need to share an HTTP session, etc.

Creating different GWT entry point modules (option 2) could solve this issue as well - there would be only one web application this time, but each module would be accessible via a different URL. The downside here, however, is navigating between these modules (e.g. inventory-to-orders or orders-to-inventory). First, the normal mechanism for navigating in GWT 2.1 (i.e. placeController.goTo()) only works within modules, and so it'll be necessary to "hard-code" the module that you are navigating to in the navigation code (e.g. Window.Location.assign()). Second, client-side state cannot be shared between GWT modules with different entry points, so if a module needs to know about something from another module, it'll have to go to the server.

These disadvantages may be acceptable, especially if the different sections are accessed by mutually exclusive sets of users, and so cross-section navigation is rare. Otherwise, keeping things together, either in one entry point but multiple GWT modules (option 3) or one module altogether (option 4), will be necessary, and you'll just have to live with the new, AJAXy-style URLs.

Compile Time

As you know, in GWT applications, Java code gets compiled into Javascript. This compilation can take a fair amount of time for larger applications (even when compiling for just one browser), and slow builds make developers less productive. Not good. Of course GWT is smart enough to only compile what's changed, and most times developers are working within the hosted mode which doesn't necessitate a manual build, but even still, in a team environment "clean" builds are sometimes justified, and if these are slow it can significantly disrupt a developer's flow.

Surprisingly, segmenting applications into multiple entry point GWT modules (option 2) can often do more harm than good, especially in the case where the shared code is large. For each entry point module, GWT re-compiles everything it needs, so if there are two modules, "orders" and "inventory", and they both share "common", then "common" will be compiled twice (note: this is true even if common is a separate project and has already been compiled into a common.jar - i.e. the JAR contains no Javascript!). In this case, segmenting into multiple entry point modules would therefore significantly hurt build time, not help.

If the application is significantly large, then finding a way to speed build times, outside of the well established configurations/optimizations, might still be necessary. Segmenting into separate GWT modules with one entry point (option 3), allows for a possible work-around. If developers typically work within only one functional module at a time (e.g. orders or inventory), it's possible for them to comment out (in their local environment only!) the references (e.g. imports in gwt.xml or code references, etc.) to the other un-used functional modules, thereby bypassing the GWT compile for these unnecessary modules in their local build. Basically, the idea is, using the example above, don't make the developer of the Orders section compile the Inventory section code when he'll only ever be working within Orders.

Of course developers have to remember not to check these files in, and of course the main build (e.g. Hudson, etc.) will build everything (e.g. Orders plus Inventory) and so will take the normal amount of time, but at least locally things would be faster. (Note: this is a trick we've used on projects with success, but I haven't heard others mention it - I'd love to hear about your tricks...or if you see a flaw in this approach).

Maintainability and Reusability

From a maintainability and reusability perspective, creating one monolithic application is a well-known path to a big ball of mud. If different teams of developers work on different sections, or if different pieces of the application may be re-used in the future by other applications, then modularizing in some way will be helpful (options 1, 2 or 3). In some cases this may mean creating one share-able set of client code (e.g. common), or possibly many (infrastructure, domain model, etc.) - depending on the expectation for re-use. Again, in general, for an enterprise-scale GWT application, it's important to put some thought into dependency organization, using either projects, web applications, GWT modules, or packages as a unit of structure.

Conclusion

As I mentioned at the start, the "right" approach to structuring a large-scale GWT application completely depends on the requirements of the project. I hope this post has at least shed some light on a few of the factors that are important to consider, but I know I've only scratched the surface...so please let me know about your experiences, or what I might be missing!

public void someMethod() {
  Service someSerive = new SomeService();
  String str1 = someService.foo();
  String str2 = someService.bar();
  doSomething(str1, str2);
}

In GWT, things are necessarily a little trickier. Using the standard RPC plumbing, calls from the client to a remote service are handled asynchronously, via GWT's AsyncCallback mechanism. For example:

someService.foo(new AysncCallback() {
    public void onFailure(Throwable caught) {}

    public void onSuccess(final String str1) {
      // do something here;
    }
});

This is relatively straight-forward. If we wanted to make two calls to some service, however, then do something after both have completed, our first instinct might be to just chain the service calls - e.g. when one service completes, call the second:

public void someGwtClientSideMethod() {
  SomeServiceAsync someService = GWT.create(SomeService.class);
  someService.foo(new AsyncCallback() { 

    public void onFailure(Throwable caught) {}

    public void onSuccess(final String str1) {
      service.bar(new AsyncCallback() { 

        public void onFailure(Throwable caught) {}

        public void onSuccess(String str2) {
          doSomething(str1, str2);
        }
      });
    });
  }

While this works, it's not ideal. First, it's really ugly...and this is just for a case with 2 chained async calls - imagine if you had 3 or 4! Second, it's slower than it needs to be, since the calls are made serially rather than in parallel.

One obvious solution is to just combine the two service operations into one (an argument for this here) - only one call is made to the server, and you don't have to bother with the ugly nesting:

public void someGwtClientSideMethod() {
  SomeServiceAsync someService = GWT.create(SomeService.class);
  someService.fooAndBar(new AsyncCallback() { 

    public void onFailure(Throwable caught) {}

    public void onSuccess(final FooBarResult fbr) {
      doSomething(fbr.getFooResult(), fbr.getBarResult());
    }
  });
}

This is better, but unfortunately combining service operations like this isn't always possible (or practical). Services may not be under your control...and even if they are, it's a lot of work to aggregate operations, and could result in a pretty complex/messy service if there are many such combinations.

Another possible solution would be to make the calls in parallel, and then only doSomething() when both service calls return. I developed two classes, ParallelCallback and ParentCallback, for this exact purpose. Using these, the code in the client would look like this:

public void someGwtClientSideMethod() {
  SomeServiceAsync someService = GWT.create(SomeService.class);
  ParallelCallback fooCallback = new ParallelCallback();
  ParallelCallback barCallback = new ParallelCallback();
  ParentCallback parent = new ParentCallback(fooCallback, barCallback) {
    public void handleSuccess() {
      doSomething(getCallbackData(0), getCallbackData(1));
    }
  };
  someService.foo(fooCallback);
  someService.bar(barCallback);
}

Without getting too mired in the implementation guts, essentially each service method has its own ParallelCallback, which is registered with a ParentCallback. When the service has completed, the ParallelCallback then informs the parent it's done, and when the parent has heard back from each of is children, it calls its own handleSuccess() method, overridden and implemented by you. Return data from the individual service ParallelCallbacks can be extracted using the getCallbackData() method.

While this solution definitely still has some conceptual load, it eliminates the double-AsyncCallback-nesting, so seems a little simpler (to me!) than the chaining solution above. Also, as mentioned, RPC calls are made in parallel, so the overall latency should be less.

Anyway, the code is available here and here if you're interested. Let me know what you think, or if you've found another better way to solve this issue.

Note: this issue was documented on StackOverflow, and there are some good insights there as well (like check out gwt-dispatch for one).

And while this role-based model is effective, intuitive, and easy to implement, unfortunately, in most enterprise applications, it isn't enough - there are some portion of authorization requirements that can't be neatly codified in simple user-role-permission relationships.

In this post, I'd like to take a stab at defining some of the tricky (but common) authorization requirements that stretch the boundaries of basic role-based access control. My hope is that by identifying (and putting a name to!) these different types of authorization rules, we'll be in a better position to determine the authorization solution that best fits our needs - whether it's a simple role-based approach or a complex policy engine (e.g Oracle Entitlements Server, Cisco's Policy Manager, etc.).

Ok, here goes:

Subject Attributes
Beyond roles, authorization rules often needs to consider some attribute of the person to determine whether they have access (see ABAC). For example, if the requirement is: "only customers in the Pittsburgh region can approve a document", then a customer's region would be a relevant subject attribute to the authorization rule.

Attributes can be one-dimensional, in which case they can easily be encoded as simple roles (e.g. a user from the "Pittsburgh" region could be put in a "Pittsburgh" role). When multiple attributes come into play, however, things get tricky. For example, if both "region" and "customer priority" are relevant attributes, then encoding these in terms of roles would yield a messy combinatorial cross-product:

This obviously gets untenable from an administration perspective - i.e. what happens when we add a new region? Or a new attribute? For this reason, subject attributes typically need to be managed outside of roles.

Object Attributes

<

To make things more complicated, sometimes object attributes and subject attributes need to correlate - for example "only customer service reps from the Pittsburgh region can remove charges from customers from Pittsburgh" (see above). In this case, the subject attribute (customer service rep's region) needs to match the object attribute (customer's region).

Row-level Filters
Some authorization rules transcend binary permit/deny decisions; they instead state that results of a request be filtered to only those rows a user is authorized to view. For example, perhaps a doctor querying a patient database should only be shown those patients for whom he's had contact, and hide the others. Or "users from San Francisco can only view customers from San Francisco". I touched on this distinction between access control and filtering a bit more in a previous post.

Column-level Filters

User Interface Show/Hide
Protecting an action by itself is sometimes not enough - we often want to prevent a user from trying to invoke it altogether. For example, if a user doesn't have access to delete an account, then a "delete" button shouldn't be displayed on the UI. This is something that , as developers, we often forget - we assume that by protecting the service boundaries (urls, service operations, etc.) we've done enough, until we talk to users and they ask "if I'm not able to delete an account, why do you even give me the option?". Ugh.

To make matters more difficult, the logic that protects the action should ideally be the same logic that controls whether the button is shown or hidden - duplicating the logic in the client and server layers would be sub-optimal.

Date/Time
Resources or actions can also be restricted based on the date or time - for example, "users can only submit requests during business hours of 8am-6pm" or "users can only approve documents after the 15th of the month".

Expiration
There are times when a permission for a user should expire after some period of time - for example, "Joe has the permission to do X, but only for the next week". These are similar to date/time rules, but subtly different.

Delegation
Similarly, it is sometimes necessary to transfer one user's permissions to another. For example, "Bob will be gone on vacation for 3 weeks, so delegate all his permissions to Ed". In a simple role-based model, this is relatively trivial - just copy roles from Bob to Ed. When using attributes, however, it can get a little trickier.

Hierarchies and Inheritance
For easier administration, roles can be organized in hierarchies, such that the permissions of a parent role are inherited by a child role - e.g. "a super admin can do everything that an admin can do, plus X, Y, and Z". Similarly, resources can also be organized hierarchically, e.g. "the admin can access any page under /admin/".

Impersonation
In some cases, it's imperative for one user to enter a a system as another - similar to the "sudo" command. For example, it might be helpful for a customer service rep to be able to log in as a customer, in order to see exactly what the customer sees, and help debug problems. To complicate things, the customer service rep might only be able to view while logged in as the student, but not update any data.

Conclusion

As you can see, authorization rules come in a variety of flavors, and it's important to understand which your business needs, so that you can design/implement an appropriate solution. Implementing an ill-suited authorization framework, or as is more common, just implementing authorization logic as custom application code, will be a big problem later on in terms of complex, unmaintainable code, difficult administration/provisioning, and incomplete auditing and reporting.

In this post I'd like to describe the decentralized, "every man for themselves" model of authorization that is most common in organizations, and the problems inherent. In later posts I hope to touch on some possible paths out of this chaos - the XACML standard, entitlement management, and attribute services. Here goes...

The Authorization Status Quo

In any decent-sized organization, there exists a number of different systems that a given user may interact with - think HR, CRM, financial apps, knowledge management portals, ERPs, etc.. While these systems may differ along any number number of dimensions (size, technology, hosting model, etc.), each will inevitably share the common feature of authorization - in other words, each must define some rules for determining what a user can and cannot do within.

The problem is that while each system must "do" authorization, the way in which they actually do it can differ greatly. One system may take a simple role-based approach leveraging LDAP groups, a second may use an access control list tying permissions directly to user IDs, and a third may just use some internal database connecting users to roles to permissions. Essentially, authorization in the enterprise is often a wild west - there are no laws; systems do as they do, with no connection between them in terms of either approach (RBAC, ABAC, etc.), tools (e.g. JAAS, Acegi, etc.), data store (LDAP, database, flat-file, etc.), or process (e.g. provisioning/de-provisioning requests, auditing, reporting, etc.).

And while this inconsistency can be beneficial to the extent that it allows each system to implement authorization in the way that is easiest or most effective within that context, across the organization as a whole, managing authorization separately in each system is a serious bane, incurring significant costs in terms of time, resources, and quality. These costs are most evident to the organization in the areas of administration, development, reporting, and auditing.

Administration

Any system which manages authorization must provide some mechanism for allowing administrators to change access privileges for users, whether by using a nice GUI, editing a configuration file, or directly manipulating rows/columns in an underlying database. As a user's status changes (e.g. they get promoted, quit, take on new accounts, etc.), these tools must be used to re-provision permissions accordingly. Depending on the number of systems, however, this can be an extremely difficult, time intensive, and error-prone process. A simple resignation of a single employee, for example, could necessitate the re-provisioning of this user's access across dozens of different systems, executed by a handful of different administrators.

Development

For systems developed and maintained internally, authorization, like other cross-cutting concerns (e.g. authentication, logging, etc.) should ideally be implemented using a similar set of tools and patterns in order to eliminate duplication of work, maintain quality, and enable developers to translate knowledge or skills from one system to another. Unfortunately, in larger organizations, this is often not the case: authorization is implemented differently for every application, using the approach, patterns, and tools selected by the development team, usually with no concern for commonality or re-use across the organization. Whether a new project uses Spring Security, AzMan, or some other custom solution typically depends on the whims and proclivities of the developers, and the result of reinventing the wheel like this is higher development cost, lower quality, and discrepancies in feature set and usability (e.g. fine-grained authorization may be possible in one system while another has a one-size-fits-all/none model).

Reporting

A generally accepted principle of authorization is that of least privilege, giving users as little access as possible to perform their job. In complex enterprise environments, where users are in the thousands, upholding this principle is crucial both from a security perspective as well as a regulatory one. And although this seems simple in theory, in practice, when systems all have their own model for authorization, it's easier said than done. The reason is that before we can even ensure what user X should be able to do, we need to first understand what he can do now. To this end, generating a simple report detailing what permissions a user has is crucial. Unfortunately, this simple view is not trivial to produce, entailing the manual (and collaborative task) of going to each system, discovering what the user can do, and then aggregating the individual sets of permissions into one global report. In other words, there is no one report that can tell "what can Joe Smith do across the organization".

Auditing

Finally, auditing is another key component of authorization, both from an administration perspective (e.g. "Joe granted permission X to Sandy on 7/20/10") and an enforcement perspective ("e.g. Joe was authorized to approve order 1234 on 8/1/10"). Unfortunately, auditing of this kind is implemented inconsistently at best or not at all at worst. Piecing together how a user's permissions changed or what a user did on any given day, across the organization, with a model of decentralized authorization, is virtually impossible, necessitating scouring through different log files, database tables, etc.

Conclusion

These are just a few problems of implementing authorization independently across different systems in an organization. In later posts I'd like to explain how steps can be taken to centralize some elements of authorization, to the benefit of administration, development, reporting, and auditing. As we all know, there is no silver bullet - every decision comes with a set of trade-offs, however given the status quo of decentralized authorization, it's clear we can do better!

It got me thinking about how in any field (whether it be software development or biology, finance, etc.) people have very finely developed intuitions about what is "right" or "wrong" - something that seems akin to moral judgement. This happens in software development without a doubt - people who have been in the field for a while can have very powerful, immediate, gut reactions to problems - they just know that a certain path is "right" or "wrong".

The question I'm kicking around is, in software development, how are our "moral" intuitions best formed - by rational deliberation (e.g. reading, observing, studying, listening) or personal experience (playing many roles, living in the trenches)? In other words, which route would lead to greater "moral wisdom" - more study or more lived experiences?

(This question is a lot like the kerfuffle about Justice Sotomayor's "wise latina" comments months back - is she more "wise", coming from a rough neighborhood in the Bronx, than someone who came from the upper-middle class 'burbs?)

My sense is that no amount of study or observation can make up for living in the trenches and feeling the pain of bad decisions (your own and others'!). I can read all day about why unit testing is great, but it doesn't sink in to the level of "moral intuition" until I live through an experience where there's a million lines of code, no unit tests, and so no clue as to whether your change broke something else or not.

Anyway, just a thought...

I may work on module A this iteration and you on module B, but next iteration I may work on B and you on C.

No developer has sole responsibility for any piece of the system, we all own it together. If I need to update some piece of the system to implement my assigned user story, I can do it, with the caveat that unit tests should pass and probably some quality check should be met (e.g. peer review, etc.).

In any non-trivially sized team, this is probably a necessity; the alternative model, strict ownership of code, will just yield unnecessary gridlock, turf-wars, and code ghettos as developers operate unchecked in their own silo, hording knowledge, and protecting the sanctity of their individual assets at the expense of the overall architecture.

And even knowing this, that individual code ownership isn't really feasible in an organization, I still think that when we accept the model of collective ownership, something important is lost: craftsmanship.

Sure, Agile advocates would argue that pride of ownership just extends to the work of the team rather than just that of the individual. To some degree this does exist, but it's not the same. When we have ownership of something, whether a design document, an API, or a UI screen, it's our reputation on the line, our creativity and execution that make it a success or failure. We make decisions on our own, quickly, without having to reach team consensus, for good or for ill, accepting the consequences. And when it does succeed, we derive a strong satisfaction from knowing that someone found value in what we alone designed and created. We helped make something better, easier, faster. The freedom of owning our work is what makes programming fun...and meaningful.

Without ownership, this satisfaction is muffled. When people collectively own everything, no one owns anything in particular. Collective ownership often means collective design, and so there's less investment in the overall success, because, hey, these aren't necessarily my ideas, they're the group's. And because everyone's a stakeholder in all the code, decisions are often made via sophistry and compromise, producing an end design that often looks more like a piece of legislation and less like a piece of craftsmanship.

It reminds me of a favorite quote of mine from Steinbeck in East of Eden:

I know, to argue against collective ownership in the era of wikipedia, social media, and open source is probably insane. Working collaboratively obviously can produce high quality results, and benefit the collaborator as well.

Even still, talking to colleagues and friends that do software development on teams, I see this unsatisfied need to own. Just about every programmer has dreams about their own side project, where he makes the decisions, where he is responsible. It's this freedom which most of us need, and which produces works of craftsmanship.

So, toward the goal of evoking usability requirements, here are 6 important questions to ask your users or business owners:

1. Should the back button be supported and in what capacity?

By default, to the end user, a GWT application will just be one URL (for example, www.myshoestore.com/index.html), and so clicking “back” will just redirect the browser to the previous site visited (which is extremely frustrating). For most applications, this won’t cut the mustard – users will need to keep their trusty friend the back button in some capacity.

The good news is that GWT provides a very simple mechanism, via the History class, to append tokens to the end of the URL (e.g. “www.myshoestore.com/index.html#FindShoe”), in effect creating virtual “pages” or navigation points within the application. Clicking back, using this solution, will revert the browser to the previous token, not the previous site (although it’s on you, in your code, to refresh the state of the application to correspond with the changed token).

The tricky part here, in my experience, is not the implementation, but rather defining with your users where these navigation points are within your application. Remember that this is RIA, so anything’s possible – navigation points can be as fine grain as needed. For example, a user could expand a node in a tree widget, click back, and the app could “un-expand” that node. This might be unnecessarily fine-grained, but it is possible. Most likely, however, navigation points are more coarse grain, for instance different “views” in your application (like in Gmail with the “Inbox” view, the “Compose” view, etc.).

Whatever the decision, it’s critical to define where these points are up front so that they can be built into the application framework.

2. Which screens can be bookmarked?

It’s often the case that users will want to share “places” within your application with others, and while coding for back button support (see above) will take you a step in that direction, it won’t get you all the way there. For instance, if a user does a search for “Nike” and the token “#SearchShoe” is added to the url (e.g. http://www.myshoestore.com/index.html#SearchShoe), when the user shares this URL with a friend, there will still not be enough context in the URL itself to let the application know to load not just the “Search” page but also with the “Nike” result set.

The solution here is to use tokens in the same way we used the query string in traditional web applications – so what used to be http://www.myshoestore.com/search.jsp?shoe=Nike in GWT could now be http://www.myshoestore.com/index.html#SearchShoe-Nike. Again, using the History class, the token “SearchShoe-Nike” would be read, parsed, and the appropriate search result would be loaded. Again, like with back button, the difficulty is not necessarily in writing this code, but understanding up-front which views in your application are “bookmarkable”, and ensuring that your framework supports this.

3. Which widgets will you support?

GWT out of the box comes with a rich set of widgets (trees, grids, menu bars, etc.) that alone will be a huge improvement over the clunky, static HTML components of the traditional web application (think text boxes, buttons, checkboxes, etc.). Oftentimes these will be all you need, but in some cases users will want more.

Fortunately, a number of very utilitarian, high-quality, and flashy open source and commercial widgets are available – things like calendars, charts, or drag-and-drop, etc. Beware, however, because this extra UI bling comes at a price. Off-the-shelf widgets can degrade performance, add application complexity, decrease UI consistency, and cause license problems. In my experience, it’s important, up-front, to assess the general UI needs of the users, identify the definitive set of widgets that will be supported within the application, and then stick to it.

In many cases, trade-offs will have to be made. For example, in a recent project, despite its power and pizzazz, we decided against leveraging GWT-EXT since we perceived the costs to complexity and performance to be too high for our relatively small application. Again, designers, users, and even developers will probably want to take an ala carte approach (a widget from here, a widget from there), but it’s important to keep the big picture in mind, and get everyone on the same page as soon as possible with respect to which widgets are in and out of bounds.

4. Is Internationalization a requirement?

GWT has excellent support for internationalization, but as with any other application, this is a non-functional requirement that is absolutely crucial to know up-front. Externalizing presentation text after the application has been built is time-consuming, risky, and generally not good for your constitution.

5. How will input validation behave?

Most every enterprise application validates user input to some degree, and as any developer knows, this can be done in any number of different ways. For example, in traditional web applications, it was most common that a user would enter all of the data, submit the form, and then the system would check the input and inform the user of any validation errors. Obviously with RIA a richer set of interaction patterns are possible – for instance, data can be validated “just-in-time”, as the user enters it, rather than waiting until the entire form is submitted.

Whatever the right interaction pattern is for your application, it’s important to define this up-front as well, and stick to it for all screens. Knowing how validation should behave will help you pick or design the right validation framework (GWT does not come with validation out of the box but there are open source libraries), and will help ensure that input is validated consistently across your entire application. Failing to do so, and therefore letting developers determine how to validate input on their own screens, is a recipe for complexity, redundancy, and inconsistency.

6. What happens on asynchronous calls?

GWT is obviously based on an AJAX model, making many asynchronous calls via Javascript rather than synchronous page loads. It’s important to know, however, the expectations of the user while these asynchronous calls are being made. Is the entire application frozen until the call returns? Or is the converse true, and the user can still some pieces or components? Which pieces? Should the user be able to cancel an asynchronous call (and what would this mean)? Should a “Progress” bar be shown? And so on.

Because asynchronous calls are the backbone of a GWT application, it’ll be helpful to understand their usability requirements early. In a previous project, we implemented a custom AsyncCallback class that showed an “in progress” modal dialog box whenever an asynchronous call was made to the server. This obviously wouldn’t suit all needs, but it worked for our users.

Conclusion

In conclusion, while definitely not an exhaustive list, I hope these questions at least sparked some thought as to what types of things you’ll want to ask your users and business owners prior to jumping into your GWT code.

And of course I’d love to hear any suggestions or experiences you have. What usability questions would you suggest asking users up front? Please share. Thanks!

At an aggregate level, however, code metrics are seldom so helpful or straight-forward. When seeing that a source tree has 160,000 lines of code or an average cyclomatic complexity of 4.12, our first thought is usually "interesting!", followed shortly by "well, now what?".

The problem is, in my experience, we often look at our code metrics in isolation, without good comparison points, leaving us to wonder whether the numbers we see are big or small, typical or abnormal, good or bad. In the end, it's not clear what to do, if anything.

For code metrics to have any real strategic value, whether it's justifying a major refactoring effort to the executive team or just proving that the recent adoption of some new practice (e.g. code reviews, automated testing, etc.) actually had the salubrious effect that you said it would, they must be looked at within some broader context. And there must be a purpose. Otherwise they're just interesting numbers, no more no less.

So, to that end, here's a very simple, low-overhead approach I've taken on a few projects, with some success'

Define a Purpose

• What is the pace of development? Are we moving slower or faster than before?
• Is the code inherently difficult to maintain? Are there modules that are hopeless?
• Should we expect greater or fewer bugs in this iteration? In general, is quality improving or getting worse?

Pick which Code Metrics to Use

Once it's clear the questions you're trying to answer, the right metrics and tools can then be picked to help build your case. Here are a few metrics and tools (Java-based) to consider, but there are plenty more:

• Lines of code (JavaNCSS, Cloc)
• Code complexity (JavaNCSS, PMD, CheckStyle)
• Unit Test Coverage (Cobertura)
• Copy-paste instances (PMD)
• Style violations (CheckStyle)
• Design violations (PMD, CheckStyle, FindBugs)
• Possible bugs (FindBugs)
• Number of check-ins

The important thing, again, is that these metrics map to some business question you're interested in. Less is more, in my experience - it's very easy to get lost in a sea of code statistics, so spend your time only on numbers that could have some practical impact, avoiding those that are inexact, nebulous, inaccurate, or just generally not useful.

Track Code Metrics over Time

Very often your question is time-oriented - i.e. "are things getting better or worse"? If that's the case, then metrics can be tracked over time, either by iteration, sprint, or release, or by day, week, or month.

And although there are a few tools that do this aggregation for you (metric_fu for Ruby, QALab or PMDReports for Java, etc.), I've found that few give me exactly what I need with little setup or configuration hassle. In my explorations, I've found it easier to just use the tools I need separately, and aggregate the metrics (dare I say it!) manually in a spreadsheet. For example, two metrics I often keep track of are lines of code and average cyclomatic complexity.

For calculating the lines of code I use a nifty tool called Cloc (which take no more than 5 minutes to set up):

  C:> cloc codebasemyprojsrc

...which gives me something like this:

For the complexity statistics, I use JavaNCSS plus a simple program I wrote to filter out the getters and setters (since these can significantly skew obscure complexity numbers, making your system look a lot simpler than it probably is!):

  C: javancss -function -recursive c:codebasemyprojsrc | java FilterGettersAndSetters

...which gives me this:

Setting up these tools (downloading, putting things in the path and classpath, etc.) takes maybe a half an hour, and then each time I run them, just seconds. Typically, I'll do this at the end of every sprint, put the metrics in a spread sheet, and then generate a handy graph that I can send to the team or use in a strategy discussion or status report.

Note that without the context of time, seeing a complexity score of 3.5 alone doesn't mean much. Knowing, however, that complexity increased from 2.4 in the last sprint to 3.5 in this sprint does - it means something happened, whether too much functionality was fit into the most recent sprint resulting in a slip in quality, or perhaps a new junior developer joined the project, and probably needs to have his code reviewed.

Compare Code Metrics across Projects

Another way to give your metrics meaning is to compare them against other projects - whether internal to your organization ("look, project A has 12% unit test coverage rate whereas project B has 78%!") or even with other open source projects ("our project has an average cyclomatic complexity of 4.57, but Spring, Hibernate, and Struts all have less than 2.5"). Although your sample size probably won't be statistically significant, it can give a good rough benchmark to help you determine what's acceptable and what merits concern.

Note: these are just hypothetical numbers to show how metrics could be used! Again, with little effort, these metrics can be collected manually using simple static code analysis tools, as long as you know what you're looking for.

Conclusion

In the end, code metrics alone won't tell the whole story, but used thoughtfully and purposefully, they can be invaluable when making strategic product or project decisions - for example, helping to illuminate maintainability problem areas in your architecture, predict where and when quality slippages would be expected, and, in general, determine whether the ship is going in the right direction or not.

Anyway, I'd like to hear about your experiences with using code metrics. Have aggregate statistics helped in any way? Which ones? How have you used them? And which tools have been useful?

In this first post, I’d like to take a quick trip through some of the basic terms and definitions of authorization, laying the conceptual groundwork for the later posts. Ok, let’s get started!

Access Control Policies

In general terms, an authorization or access control policy defines whether a given subject can perform an action on a particular object. For example, a policy could determine whether "Hines Ward" (subject) can delete a charge (action) from "Mike Tomlin's" account (object). (*2) This decision, to grant or deny access to a specific object or resource, is commonly based on one or more the following four factors:

1. Role-based (RBAC): Most often, access to a particular operation is granted by virtue of a specific role that a subject is affiliated with. For example, the ability to remove a charge could be granted to any user with an "Account Admin" role.
2. Attribute-based (ABAC): Access control may also be based on some inherent attribute of the user or object. For example, Account Admins from the Northeast region (user attribute) may remove charges from customers from the Pittsburgh district (object attribute). In this way, access control policies define a relationship from subject to object that must exist in order to perform the action.
3. Consumer-based: In an SOA world, service endpoints may be open to the network, and so it may be necessary to manage access to an operation based on the consumer of that service. For example, ensure that only the Customer Service may invoke operation "getAccountInfo(customerId)" on the Accounting Service.
4. Time-based: It is sometimes necessary to restrict access to a particular resource based on the time of day or calendar date - for instance, "only allow changes to accounts during normal business hours, 8am - 5pm".

Access vs. Filtering

Beyond the subject-action-object form of access policy defined above, there's another type of authorization that's just as crucial in enterprise applications: response filtering. Although both types are based on the authorization factors defined above (role, attribute, consumer, time), filtering policies determine not whether a user be restricted to a specific operation on an object, but rather that the data that is retrieved from an operation be narrowed to only those records or fields that the subject is authorized to view . For example, when user "Hines Ward" of the "Pittsburgh" region calls operation "fetchAccounts()", the system can allow access, but return only those account records for customers in the "Pittsburgh" region, and not account records for customers in other regions.

In my experience, these two types of authorization ("access" and "filtering") are often mistakenly conflated, and so are assumed to be satisfied by the same authorization framework. Unfortunately, this isn't usually possible - a framework or solution that protects access to a resource (e.g. URL, endpoint, etc.) is fundamentally different than one that filters result sets or message bodies. It's important, therefore, to identify the type of authorization requirements that exist to guide downstream design.

Architectural Components of Authorization

Having trudged through some of the conceptual territory of authorization, we can now dig into some of the tangible, implementation details. In any authorization solution, most of the following architectural entities exists in some form or another, though they may not be clearly abstracted:

• Policy Definitions: Where policies are actually defined and stored. In the case of simple RBAC models, this may just be a database managing relations between users, roles, and entitlements, but in more complex ABAC models, these could be separately defined policy files (e.g. XACML). Note that policies are also quite often found "hard-coded" in business logic or UI code (e.g. "if(user.group == 'admin') showButton()"). (*3)
• User Store: The central repository for core user information (e.g. name, id, etc.) - think LDAP for enterprise apps, or perhaps just a "user" table in a RDBMS for smaller apps.
• Attribute Store (i.e. Data): As I defined it in the section above, authorization attributes (whether subject or object) are just plain old business data that happens to be relevant to determining access for a user. For instance, in the example above, the customer's region is a relevant authorization attribute, and would likely just be stored as a "region_id" column on the "customer" table.
• Policy Enforcement Point (PEP): The place in the architecture that enforces access control. For example, in a simple web-application, this could be a servlet filter that protects the resource.
• Policy Decision Point (PDP): The place in the architecture that makes access control decisions. Note, in smaller applications, this likely is the same as the PEP, but in larger, distributed applications, it could be different. Also, note that in some cases Policy Definitions are embedded within the Policy Decision point, but may be externalized (e.g. XACML).
• Response Filter: The entity responsible for winnowing a data set down to only those records a user is authorized to view.

A "Simple", non-SOA Example

Already, authorization seems a little harrowing - and we haven't even gotten to SOA yet! Before we do, as a final step toward laying the authorization groundwork, let's map some of the concepts and architectural elements defined above to territory that's most likely familiar - a simple, stand-alone web application.

As the diagram above illustrates, authorization in this "simplest" case can still be rather complex. Policy definitions and decision points are scattered throughout the different layers of the architecture - an interceptor responsible for protecting resources (i.e. URLs) based on roles, the UI for showing/hiding information based on permissions, the business logic for executing complex, attribute-based authorization decisions (e.g. customer can only view their own account), and the data access layer (or database) for filtering data sets to only those records the user is authorized to view. The database stores user, role, permission, and attribute data, queried by the different PDPs.

What's Different in SOA?

As I've written about before, SOA puts a new twist on old problems, and it's the same for authorization. What was a delicate but well-traversed and tractable issue before, is an extremely complex and risk-ridden problem when you stir in the ingredients of a SOA-based enterprise architecture - loosely-coupled services, registries, process engines, stand-alone user interfaces, ldap directories, etc. Although I'll cover these pitfalls in future posts, here are a few monkey-wrenches SOA throws into the authorization machine:

• Is there a central repository for all users of an enterprise?
• Is there a central repository for roles and permissions for all services in the enterprise? If not, does each service maintain its own unique set of roles and permissions for users?
• How are authorization attributes propagated between services?
• Can policies be managed centrally, such that changes to access control necessitates only going to one place, and not to every individual service?
• What happens when a policy depends on retrieving data that is not local to the service it resides in? In other words, can policies call other services to retrieve authorization attributes they need to make allow/deny decisions?
• How can a service ensure that only other services call it?
• To what extent can policies be modified at run-time? How can they be managed?

And this is just the tip of the iceberg! In my humble opinion, most of these issues boil down to the same central vs. local diametric decision - i.e. what authorization components should be centrally shared by all services (e.g. User store/service, etc.) and which should be locally (read: redundantly) defined, within each service. Obviously, there are major trade-offs on both sides - performance, maintainability, scalability, and reliability to name just a few.

To sum up, I hope I was able to lay some of the building blocks of authorization. In later posts, I'd like to touch on different approaches, patterns, products, and standards for authorization in SOA. I'd love to hear what you think so far, so please feel free to drop me a comment!

Endnotes

1. For those people who get these mixed up (I used to be one!), authentication tackles the problem of verifying that the user is who he says he is, and identity management with the multi-faceted problem of how users are given identities and how these identities are propagated. Authorization, on the other hand, assumes these first two issues are solved, and deals with ensuring that a user may do only what he has permission to do.

2. Yup, I'm a Steeler fan:)

3. The dividing line between "business logic" and "authorization policy" is often fuzzy - and it's not always clear whether a requirement should be implemented within the confines of the authorization framework, or whether it can be happily embedded within normal business logic code. In an email comment from Rick, he suggested that a good litmus test is the requirements around manageability - e.g. who is responsible for managing access rights, does the logic need to be configurable at run time, etc. I think this is a great point.

Types of Properties

Before designing an approach to managing property files, it's critical first to understand the types of properties are to be managed. Here are four characteristics to consider:

• Are some properties environment specific – in other words, do they have a different value in one environment than another? For example, a database connection string would differ in the TEST environment than it would the PROD environment.
• Do any properties contain sensitive information, like passwords or credentials, that can’t sit in plain text in the deployment unit (e.g. WAR, EAR)?
• Do any properties need to be located external from the deployment unit (e.g. on the file system rather than in the WAR, perhaps to be managed by a systems administrator)?
• Should properties be dynamic, in the sense that they can be updated at runtime (i.e. without requiring a restart of the application)?

Each of these special characteristics adds a degree of complexity to applications - necessitating additional infrastructure beyond the simple Java Properties class.

Spring’s PropertyPlaceholderConfigurer

Fortunately for us, the Spring framework, with the PropertyPlaceholderConfigurer and PropertyOverrideConfigurer, provides the features and hooks we need to manage these cases (with the exception “dynamic” properties). And true to its philosophy, Spring makes simple things simple, and complicated things possible.

The PropertyPlaceholderConfigurer allows properties to be pulled in to the application context file. For example, in the simplest case, a property “db.user” defined in database.properties could be pulled into the ${db.user} placeholder:

<bean id="propertyPlaceholderConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
 <property name="locations">
 <list>
 <value>classpath:database.properties</value>
 </list>
 </property>
</bean>

<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
 <property name="user" value="${db.user}"/>
 <property name="password" value="${db.password}"/>
 ...
</bean>

The PropertyOverrideConfigurer works in the opposite way, pushing properties from property files into properties in the context file (without having to specifically define a place holder – e.g. ${db.user}). These are both well documented in the Spring API and documentation.

Tips

Given the different possible types of properties, and Spring's property framework, here are a few tips for managing them:

1) Consider using a JVM System property that sets the environment of the machine (e.g. "my.env=TEST"), telling the configurer which property file to use. For example:

  <context:property-placeholder
      location="classpath:db-${my.env}.properties"/>

If the "my.env" property was set to "TEST", then obviously the PropertyPlacementConfigurer would look for a file called "db-TEST.properties". For Tomcat, this property can be set in the admin console or defined in a startup script (e.g. "-Dmy.env=TEST") - neither of which is very elegant. Alternatively, it is possible to use JNDI with Tomcat, defining "my.env" in the server.xml and the context.xml of the web app. (Note, there are of course many other ways to solve this environment-specific problem, but this is an easy and relatively straight-forward one.)

2) It may be necessary to set the ignoreUnresolvablePlaceholders to true for any PropertyPlaceholderConfigurer, which will ensure that a configurer won’t fail if it can’t find a property. Why would this be a good thing? Oftentimes, one context file will import other context files, and each may have their own configurer. If ignoreUnresolvablePlaceholders is set to false (the default), then one configure would fail if it couldn’t find the property, even if another configurer down-stream could find it (see good explanation here). Beware, however, since this will suppress warnings for legitimate missing properties, making for some tough-to-debug configuration problems.

3) To encrypt properties, subclass the PropertyPlacementConfigurer and override the convertPropertyValue() method. For example:

protected String convertPropertyValue(String strVal) {
  //if strVal starts with “!!” then return EncryptUtil.decrypt(strVal)
  // else return strVal
}

4) Consider using the systemPropertiesMode property of the configurer to override properties defined in property files with System properties. For one-off environment specific properties this can be a helpful solution, however, for defining many properties, this configuration can be cumbersone.

5) For properties that need to be managed outside of the WAR, consider using a System property to define where the file is located. For example, the property ${ext.prop.dir} could define some default directory on the file system where external property files are kept:

<context:property-placeholder
     location="file:///${ext.prop.dir}db.properties"/>

This entails, however, that this property is set for any process that leverages the Spring container (e.g. add the param to the Run Configuration for integration/unit tests, etc.), otherwise the file would not be found. This can be a pain. To circumvent, the configurer can be overridden – changing the behavior such that it looks to the external directory only if the System property is set, otherwise it pulls from the classpath.

6) Beware of redundancy of environment-specific properties. For example, if the solution is to have one property file for each environment (e.g. “db-test.properties”, “db-dev.properties”, etc.), then maintaining these properties can be a bit of a nightmare - if a property "foo" is added, then it would have to be added to the property file for each environment (e.g. DEV, TEST, PROD, etc.). The PropertyOverrideConfigurer is appropriate to eliminate this redundancy, setting the default value in the application context itself, but then the overriding value in a separate file. It's important, however, to document this well, since it can look a bit "magical" to an unsuspecting maintenance developer who sees one value specified in the context file, but another used at runtime.

Conclusion

Managing properties for an enterprise application is a little trickier than one might expect. With Spring's property configurers, however, the toughest part is just knowing what you need - the rest comes out of the box, or with some minor extensions.

Hopefully a few of these tips will be useful for you. Please let me know some of your own!

You are an architect of an online retailer. Your business owners have laid down the gauntlet. Well, not really they've just added a new requirement: they'd like to see a list of all orders for customers matching some specific criteria (e.g. city, name, registration date, etc.). For example, show me all orders for customers named "Holmes" from "Pittsburgh, PA". Further, this isn't for a static report, but rather a dynamic screen in their admin UI.

Sounds simple, right? In the old days of integrated applications, it would be - just join the customer and order tables and apply said criteria. Voila!

Ahh...but it's a brave new world of SOA, and some old tricks no longer apply. In your architecture, instead of one integrated application, you have two loosely coupled services, Customer and Order. Both services are deployed independently and have their own data - i.e. the Customer service has a Customer data store and the Order service an Order store.

The question stands then how can you answer a business question about customers and orders, given that this data spans across multiple services?

Here are four solutions I've come across:

1. Retrieve from Service: A pure SOA architecture may dictate that access to another service's data necessitates a call through that service's exposed interface - and this is where you come face-to-face with the n + 1 selects problem. In this architecture, your Customer Service would first have to query the customer data store for the list of customers matching the criteria. It would then, for each customer, call the Order service to return all orders associated with that customer. Therefore, for n customers matching the criteria, n + 1 queries would need to be executed (on top of n remote calls to the Order service!). The performance costs here are significant, and I imagine unsatisfactory in most cases.

2. Retrieve Directly: Another option would be to open read-only access to the Orders data store from the Customer service. This would improve performance, by saving the remote calls to the Orders service (and perhaps the associated data binding), but still wouldn't circumnavigate the "n + 1 selects" problem, since without a database join from customers to orders, a query of orders for each customer is still necessary.

3. Cache Locally: A third route would be to cache another service's data locally (perhaps through some pub-sub, Master Data Management solution). In the scenario above, some customer specific data could be cached in the Order store, such that finding all orders for customers living in "Pittsburgh" could be handled by the Orders service alone. Alternatively, there could be a special search service which caches both customer and order data locally, and would be used exclusively to handle queries across data stores.

4. Retrieve from Central Database: A fourth solution, not depicted on the diagram, would be to have only one master database from which both services may read and write. Doing so would improve performance and perhaps reduce complexity, but much of the loose-coupling, separation of concerns benefits of your SOA would not be realized - for example, a change to the orders table could have the effect of breaking the Customer service.

You may argue that this is a straw-man example - in the real world, these services wouldn't exist (e.g. wrong service granularity, etc.). Possibly true. However, any SOA will have multiple services, and some of these services will need access to data that spans other services, so in some form, the problem will still remain.

Anyway, thanks for reading, and I'd be very interested to hear your thoughts or solutions!