Now, just one day later, everything has changed. The entire team is confident that we’re going to complete this milestone with time to spare.

What happened?

Well, on Tuesday we already knew what the real issue was. A couple months back we decided to use a framework that was new to some members of the team. As it turned out, the learning curve was steeper than we’d realized.

Two developers are primarily responsible for the development work in this framework. Jim knew the framework when we selected it. Tim did not.

Jim had a lot of work on his plate, that he quietly likely could complete by the end of the day Friday.

Tim had less work to complete, but much of his time would be spent learning enough about the framework to get his work done. Tim was confident that, with assistance from Jim, he could get his work done on time. But the framework is not well documented, so Tim – and the rest of the team – had little confidence that he would be able to complete his task by Friday on his own.

At noon on Tuesday we hit on an idea. What if we found an expert on the framework to remote pair with Tim on Wednesday and Thursday? That expert could be located anywhere in the country, or even the world. We could possibly take out two birds with one stone. We could hit our Friday deliverable. And we could get some much-needed training on the framework for Tim.

By 5pm on Tuesday we had a signed SOW with an expert in the framework who was available to remote pair over the next two days.

And now it is Wednesday — just 24 hours later. Jim is on target to complete his deliverables for Friday. And Tim, after a day of remote pairing, is likely to complete his deliverables by the end of the day tomorrow, Thursday.

Thanks to the power of the inter-webs, a team that collaboratively thinks its way through tough spots, and management that is willing to back outside-the-box solutions.

Shut up shuttin’ up.

I don’t like unsolicited emails. Period. But as I’ve tried to get the word out about ShowMojo I’ve acquired some measure of temperance. It’s not easy getting in front of customers some times — especially the ones you can help the most.

Nonetheless, some companies don’t get it and just go way too far.

I received a mass email this morning from a service that we pay for to promote ShowMojo. I have zero interest in these emails and dutifully clicked unsubscribe. I was happy to do it. I had no hard feelings against the service and understand that some of their customers want these emails.

I was truly dumbfounded by what hit my inbox next. I’ll call it the: “Did you really mean to unsubscribe?” email. Choice quote from said email: “Was this a mistake?”

Yes. You’re so right. I accidentally hit the 5-point-font, off-white, no-underline “unsubscribe” link on your email then successfully (but completely inadvertently) navigated your intentionally-misleading “subscription preferences” webpage obstacle course. Absolutely I want you to continue to junk up my email box. Stupid me

But wait-a-minute. Quite irrationally, I asked my team if we currently use this particular service. The answer was no. Canceling was easy. And every $30 a month counts.

Oops. Stupid you.

A few months back I implemented a practice that I had considered using for years. I’d never done it because of the pushback I received from the teams I worked with. I regret not having done it years ago.

Every time a user encounters an error in our current product (ShowMojo) everyone on the development team gets an email. Not just for the Red Rails Box of Death, but for the little things that don’t work as well. The Javascript widgets that misfire, or fail to fire. And any time the system encounters an error, no matter how non-obvious it is to the user.

They may not be aware of it, but our users never encounter an error on their own. That error lands like a thud in every development inbox. It’s a thud because it’s unexpected. It’s a thud because it’s a call to action to the team: “Make it stop.”

When we first started this practice, there were dozens and dozens of errors a day. We had to categorize, tag, and prioritize them. It took several weeks to reclaim our inboxes. And we had to do this while pushing out new features. So there was a constant tug-of-war between current user experience and adding new value.

But one day came the silence of no errors. And it felt good, because we knew that’s how our users must feel  — even if they didn’t know quite how or why.

This isn’t the same as collecting errors are scanning logs, which can be easily forgotten or simply deprioritized. This might be difficult for huge products and strained teams. But this is something I’ll strive for with every new product, regardless of the initial resistance from the team.

It doesn’t feel right having the wrong answer for this question: “Do your users face your errors in solitude?”

This brings us to an apparent paradox. The larger and more novel something is, the more we need to put some estimates around it – lest it end up a railroad-side barbecue. Yet the larger and more novel something is, the tougher it is to devise meaningful estimates.

But this isn’t a real paradox. It’s just a starting point.

At the very inception of a project or product we don’t need a thorough estimate of everything. We just need a simple set of estimates that tell us what to label the tick marks on our deliverable-level cost graph. Were we to cost out our initial estimates, would we be measuring project deliverable in increments of one thousand dollars, or ten thousand, or a hundred thousand? This lets the organization and team know what league they are playing in and how much to invest in upfront estimation and project tracking.

Most things we need to do should not be difficult to estimate. Anyone who’s been around the organization long enough should know how long it takes to spin up a shared development environment, the average time to build out a screen, and the usual duration of system and acceptance testing. For the rest we use development spikes.

We use a development spike to tell us whether some technical activity is feasible and/or to tell us how long it could take to complete.

Here are some examples of development activities (and questions) that learn more about with a development spike:

• Can we implement two-way SMS in our current infrastructure?
• Can we reduce costs and time by using an open source charting package (or is it better to roll our own)?
• How difficult will it be to upgrade to Ruby on Rails 3?
• Can we do multiple file uploads without flash?
• Will it handle the load?
• How do we make these two systems talk to one another?

A spike is typically throw-away code. It’s only purpose (at least in this case) Is to give us enough knowledge to state a fuzzy estimate with confidence.

I’ve got the acid test for you.

Identify a small product that you have always wanted to build. Commission something that you could use on the job or at home. It needs to be something that requires more than just you to complete. It must require a budget, paid help, realizable value, clear goals, and a plan. In other words, scope and fund your own project.

Yes. You fund the project with your own money. That’s what makes this an acid test.

For me, for example, this something is to rebuild and productize a web-application that has become a critical part of managing the rental properties my wife and I own. We have used two different prototypes over the last two years and I firmly believe this product captures a soon-to-be-ubiquitous element of property management. So I am taking the leap.

This PM acid test assesses whether you have the skill to contemplate and manage everything about your small project (and the product on which it is based). Either directly or via delegation you will need to:

• Define your own requirements and break them into plannable features.
• Identify a MVP (minimal viable product) and keep the project aligned to that vision.
• Find and contract with the professionals who will complete the work.
• Forecast and manage the development schedule.
• Identify the technologies to use and the deployment environment.
• Vet archiitecrure and design decisions.
• Review all completed deliverables from a functional and technical perspectives.
• Balance new features versus technical debt.
• Do anything else a project manager, product manager, product owner, stakeholder or sponsor would do or delegate.

Of course you can get help on any or all of these activities. You can hire an architect to make decisions and review code. You can hire a QA professional to write scripts and run tests. But you still need to acclimate these professionals to your product and vision. And any additional hands will cut into your budget.

Even when delegating you still need to preform an informed review of all significant work products. Delegation without review is an elephant trap.

The only certainty around this acid test is that something will go badly. If you hire four contractors at least one will be a dud – you’ll pay someone to clean up that mess. Your own conception (and delivered specs) of the system will never be free of potholes – someone will need to go back and patch those, or dig up the road and start over. And your vision of the product will never be communicated so clearly as to avoid every misstep – that’s more rework still.

You will need to be on top of everything. Or you can sit back and watch Team Burn Rate blow half your budget in two weeks. Whenever anything goes wrong it will be your fault. You did not give clear requirements. You hired the wrong person. You changed your mind. You let one decision sit too long and made another decision too early. You let a technical issue sit unaddressed as it smoldered its way through your release timeline.

Can you complete a releasable version of your own, personably-funded product before you cut off the money supply because you can’t lose any more? That’s why it is an acid test.

This reminds me to ask the question: Has anyone planned for sick days in February?

I spotted a trend a few years back that I blogged about in 2009. Now I’m convinced that February is the Northern Hemisphere’s worst month for sick days. And I do account for February sick days in the delivery commitments that my teams make.

Of course, there are other perspectives. I once had a manager who, when contemplating the likelihood of illness-reduced delivery in February, rebutted: Plan at the standard velocity. That’ll encourage the healthy employees to work overtime and cover the shortfall.

He was all about morale, that one.

These deployments aren’t the one-click-then-beer variety you find on Heroku or in some seriously agile shops. These deployments are arduous day-long journeys, caravanning eight unique technologies, traversing multiple connected systems both up- and down-stream. And bad things do live in the water.

Seriously, it has been a lot of hard work and some difficult times reliving past mistakes. So here’s my list of freshened lessons that make big deployments suck less:

1. One group chat and one conference line. Don’t use email for the deployment team. It’s slow. terrible at threading conversations. And, somehow, people always get dropped off the address list – or appended to the cc field 37 times. A dozen individual chats and phone calls are terrible for anything but proving why Europe mothballed the semaphore back in the 19th century.

2. A deploy with fewer people is better. An all-hands-on-deck deploy may sound comforting. In reality. that’s just more people in the way, arguing over the best corrective path, and slowly burning out while standing at the ready. Deploys with fewer people – with clear responsibilities and fully prepared for the tasks of the day – beat mob deploys any day of the week.

3. Have your support people at the ready. Just because your core deploy team is fully briefed and prepared doesn’t mean you have all the expertise you’ll ever need. Know who else you’ll need if the fit hits the shan. Make sure that each member of this auxiliary group is ready to drop in at a phone call’s notice. Collect and distribute the best contact information for that auxiliary group to each member of your core group prior to deploy day.

4. Clear communication channel. Who owns outward communication and in what direction? People who need deploy status updates could include stakeholders, upper management, cooperate support teams, and auxiliary team members. They will not all want or need the same status reports. (Hint: you don’t want to send a 100-line deploy plan every hour to your stakeholders.) You need to decide in advance who owns which channel, the frequency of status updates, and the content of those updates.

5. Clear decision-making. When your app server and the database stop talking the last thing you need is a team that descends into bickering over which server to kick first. A deploy will involve a variety of experts, but there should be a single person in charge who – once all voices have been heard – calls the shots.

6. Clear accountability. The entire deploy team should know who is responsible for what well before the deploy begins. For me, that means one owner for each discrete part of the deploy. This separation may be by system, technology or product subteam – whatever makes sense for the deploy. Finally, only core deploy team members own deploy tasks. Sometimes a non-team-member is responsible for completing a task, but we still want a core deploy team member accountable to see that it gets done.

Requirements. Quality. Deadlines. Budgets. Scope creep.

Dependencies. Audits. Deployment plans. Disaster recovery.

Single points of failure.

Other systems. Other teams. Other everything.

Backouts. Blackouts. Timeouts. Burnout.

On call. Fire call. Five nines. Server down.

Bad managers. Weak programmers. Ill-suited customers.

The random bus.

Perhaps we hold fast to the long-term course. Fortunate, still, our leaders flare near-terms into the night. Even so our endeavor rolls on as involuntary peregrination.

Though it never was how can we return to the way it should be? Create and inspire. Band together for common purpose and tantalizing gain. Strive and fall and lift one-anther higher. Set collective effort on uncommon aspiration. How do we forget to remember? To make something awesome.

When even the development team is left scratching its head saying: “Yeah, that looks way big to us too, but … ” you know there are problems bigger than whatever comes after that ellipses.

Granted, there are mitigating circumstances. The requirements – such as they are – may delve into a new and unknown domain. The technology may be novel to both the team and the corporate IT ecosystem. But even in these cases, if the team cannot mount a strong justification for an eye-popping estimate, you have to ask whether there are other issues lurking in the shadows.

In these cases, the two most common issues I’ve set alight are:

1. The delivery team lacks confidence in its own technical ability (due to a skills mismatch, lack of training, or other reasons). This issue goes beyond the time it might take to ramp up on new technologies, adjust to a new domain, or proof a novel design approach. Teams can readily explain all this in their estimates. But an intractable problem arises – with a huge estimate in tow – when the development team fears that it’ll never fully tackle a new technology, domain, or approach. Or, sometimes, it’s an unconquered legacy technology, domain, or approach frustrating future development.

2. The delivery teams lacks confidence in its business partner’s ability to honestly collaborate and openly negotiate with the team. It’s okay if the business partner doesn’t get the requirements solidified on the first, the second or third try. Agile teams know this can be difficult, and agile processes are geared to accommodate. However, when a customer repeatedly demands prompt and accurate delivery on half-baked requirements, even an agile team will take a tanker-truck of spay foam to the delivery estimate.

While the causes are dramatically different, the end result for each issue is the same. The team’s huge estimate reflects an inability to adjust and adapt. In one case the team lacks confidence in its own ability; in the other case the team lacks confidence in its customer. Regardless of the cause, these are issues that need to be addressed, as they impede far more than the team’s ability to deliver an estimate.

Sometimes a huge estimate is just a huge estimate. Sometimes it’s something much more. Next time you see a jaw-dropper of an estimate wrapped in flimsy explanation, it might be worth taking the time to find out why.

Then, about a month back, something annoying happened. I’d search for a local joint – say Gino’s East, to order pizza – and my normal search result appeared different.

Before

After

While I noticed a difference – my restaurant listing is now a “sponsored link” – I didn’t think much of it. Then the call was answered by GrubHub and not Gino’s East. I hung up.

After further inspection, I discovered my around-the-corner pizzeria is now the unassuming red pin next to the big glaring advertisement.

The math here is pretty obvious – had I completed the call. GrubHub takes a cut from my Pizzeria. Google takes ad dollars from GrubHub. Apple, I’m sure, gets some form of compensation.

Now, let’s talk unintended consequences.

What happens if I succumb to mobile advertising Nirvana and click the default GrubHub ad on every search? I was going to order takeout from my Pizzeria anyway. Similarly, I was going to make a reservation at La Gondola on Saturday regardless of a GrubHub ad. The upshot here is that either my local restaurant takes less profit or raises its prices – one of us ultimately pays for it.

Conversely, what happens if I evade the GrubHub ad? I can – and do – click around the ad to ring my local joint directly. It’s only one extra tap. But it’s a precise tap – while cradling a baby, while walking down the sidewalk, while hanging for dear life inside a CTA train.

Perhaps the precedent is more important. I consider “Maps” an application that I paid for when I bought my iPhone. It came preinstalled. I couldn’t remove it if I wanted to. And the phone certainly wasn’t free.

It’s only one extra tap, but this is the first time I’ve been annoyed by – or ever seen – an ad in a preinstalled application on ANY phone I’ve owned. And this isn’t just any advertising. This advertising affects how I use the phone. Personally, I will pay to avoid being assaulted by advertising. That’s one reason I don’t have an Android phone. That’s the major reason I’m selective about my use of Google.

What’s the upshot?

For me, I see a behavior change. One by one, I’ll add local shops and restaurants to my contact list. Hopefully it’ll take Google a couple more years to find its way there.

For mobile usability, this is a loss, as the extra ad dollar is clearly more important than the paying customer – even on an iPhone.

For my local joints, it’s one more national player trying to take a bite out of their pie. That’s also bad for me.