Wrapping up the your product failed because you didn’t enable your users to realize value branch of the root causes of product failure, is this article on the context in which your user is using your product.  If you ignore your user’s context, they won’t be able to realize the value you provide – or won’t be interested in solving those particular problems at that particular time.

Value Realization

One aspect of solving the right problems is making sure that you’re enabling your user personas to realize value.

[larger image]

This can be further decomposed into different aspects of realizing value.

[larger image]

Previous articles reviewed

• Targeting the right users
• Focusing on the important goals for those users
• Addressing those needs sufficiently

Even when you pick the right users and the right problems, with a good understanding of what you need to do to solve those problems, you still may not enable your users to realize value – thereby not really solving their problems.  One way to think about it is that your product doesn’t implicitly solve the problem – the user solves the problem, using your product, if they can.  Recognizing that users learn, grow, and change over time is important to assuring that you are creating a product that they can use effectively.

You also need to make sure that your product is targeting the context(s) in which your users can – and want to – solve the particular problems you are trying to help them solve.

Context

*

“You keep using that word…” [Happy 25th anniversay, The Princess Bride!].  Fair point.  Context from a product management point of view, means the environment, situation, and mental state of your user.  In the user experience field, folks are trained to do contextual inquiry, essentially to help designers get a handle on the environment in which people use a product, in order to design a more usable product.  A product manager, maintaining an outside-in perspective, can do the same thing, to make sure that the capabilities of the product are the right ones.

Precious has a great presentation where they identify several patterns for designing products, taking into account that we should not think about context as the device a user is using right now, but instead to think about all of the devices that people use throughout their day.

They show that thinking about different interface form factors works as a reasonable proxy for the physical environment a person is in.

They go into more detail in their blog post, definitely check it out.

What they’ve keyed in on is that you can’t just design (or product-manage) for a persona, you also have to think about their environment.

The relevance of a persona’s goals, in a given context, are a reflection of the actions that person will take in a given environment. If you’re watching a video on a large screen, your entertainment goals may be immersion, but if you’re watching it on your phone, the goal is more likely to be distraction (to kill time).

One of the most successful mobile shopping apps, for example, was created with the intent of enabling a person, while standing in line at a coffee shop, to purchase something they already knew they needed – before reaching the counter. The primary focus was efficiently processing a transaction – oh yeah, I need a new thumb drive – not enable a window-shopping experience. The team that developed this app referred to their user as a “spear fisher” – capturing both the aspects of the persona and the contextual-relevance of a particular goal.

Getting Things Done (GTD)

As another validation-point of the importance of a user’s context as impacting which goals are relevant to users in a particular context, we can look to the popular Getting Things Done movement created by David Allen.  In GTD, a context is defined as “the tool, location, or person that is required to be able to complete an action.”  One of the ideas that makes GTD powerful is that it encourages you, when you think of something that you need to do, to “file it away” (if you can’t do it that instant) for when you can do it – this relieves your brain of the burden of juggling that to-do item, as long as you “trust” your system for filing-away and eventually doing stuff.  The system for filing away items has evolved to include the idea of the context in which you can actually accomplish something.  For example, picking up replacement furnace filters would be something that you can accomplish in the context of “while out running errands” – and your system will remind you to do this when you are out running errands.  It will not remind you to get replacement furnace filters while you’re at home watching a movie.  The emergence and embracing of this concept serves (me) as significant validation of the hypothesis that goals have relevance that varies by context.

Pulling that back into product-management, your product could have failed because while you identified an important goal for users, you didn’t take into account the context in which your product is being used – and that your product is being used in a context where solving the goal is not important – or where the user interface needs to be designed to support that particular goal.

Customer Journey Mapping (Customer Experience Mapping)

Chris Risdon at Adaptive Path has a great presentation on mapping a customer’s experience (or journey) throughout the lifecycle of engagement in a process or flow (or with a company or product).  This is a tool / technique principally developed for informing user experience (e.g. design decisions).  However, I have successfully co-opted some aspects of journey mapping to frame an outside-in, customer-centric view of user goals.  The journey spans the breadth of engagement in an experience, broken up into different steps.  Each step reflects a shift in context and it becomes apparent that different goals matter in different contexts.  Some goals exist in multiple contexts, but with differing relative priorities, and often with nuanced differences of what it means to address the goals adequately.

If you’re dealing with a single (or simple) product, you may not get value from building out a journey map (and it is a non-trivial investment of time).  If you’re looking across a portfolio of products, or looking for opportunities to address “adjacent” problems – this can provide a great framework for exploring and contextualizing (yup, I went there) the decisions you’re making.

Summary and Attributions

*Thanks Lane Halley for the image of people in chairs.  Not sure where I found it, but here’s a link to a great presentation on UX for lean startups from Lane.

Make sure that you’ve got the context in which your users will use your product in mind when identifying the problems that your product will help them solve. Without that understanding of context, you risk creating a product that does not enable your user to solve the problem it was intended to solve.

Next up in the series on the root causes of product failure – products that fail because you have ignored the user’s level of experience.  The first time someone uses your product, they don’t know anything about it.  Did you design your interfaces for new users?  After they’ve used it for a while, they get pretty good at using it.  How much do you think they like being forced to take baby steps through a guided wizard now?

Why Do Products Fail?

Your product launched a year ago.  People raved about how easy it is to learn to use your product.  Someone posted a video of a toddler using it, it kicked off a meme, you got a nice bump in downloads, and you were thrilled.  Now, people are complaining about how impossible it is to actually get anything done with your product.  And your main competitor is making hay with their message of cost-effectiveness and efficiency.  Sales have dried up to a trickle.  What the heck?!  You loosen your collar and walk in to explain to your investors why they will be lucky to get half their investment back – much less the ten-bagger your confidently forecast for them eleven months ago.

It sure is good you’re performing this thought experiment before you even build the product, instead of running the gauntlet a year after it launched.

There are many reasons a product can fail.  One way to think about your responsibilities as a product manager is to prevent all of them.

[larger version]

In this series, we are exploring as many of them as we can.  In your exploration of the reasons products fail, you started by making sure you’re solving the right problems.

[larger image]

As a firm believer in being user-centric, and making sure you’re delivering value to your users, you’ve doggedly pursued exactly what that means.

[larger image]

You’re confident that

• You’ve picked the right users for your product
• You’ve identified the problems that are important to the right users to solve
• You’ve figured out what you need to do to actually solve those problems sufficiently for those users

What you failed to do was understand that your users change over time.

Kathy Sierra’s Canyon of Pain

I wish Kathy Sierra was still writing online.  I first came across her canyon of pain model in 2007.

On the other extreme is Apple’s iMovie. It gives you almost no control, but the payoff is high right out of the shrinkwrap. It exceeds my expectations of pain-to-payoff. But pretty quickly, anyone who gets into iMovie–and is bitten by the movie-making bug–starts wanting things that iMovie doesn’t let you control. So… Apple says, “not to worry — we have Final Cut Express HD for just $299″. The problem is, the learning curve jump from iMovie to Final Cut Express is DRASTIC. There needs to be something in the middle, to smooth that transition.

Kathy Sierra, How much control should our users have?

While she uses the above example to describe a market segmentation opportunity, it describes not just the difference between users, but the changes a user goes through over time.  Taking an aerial view of Kathy’s Canyon it would look like this:

Most users are competent users.  It makes sense – you start out as a new user, until you develop competence, and possibly develop expertise (although few become experts).   How many users will use your product for long enough- and invest themselves enough in learning to use your product to truly develop expertise?

Conceptually, you need to build a bridge across that canyon, for your product to meet the needs of your users as they develop confidence (and for the few who become experts).

Learning Curves

This is an artful dance – designing for the competent users – and your personas are likely to represent competent users, since competent users will be focusing on using your product to solve their problems, not trying to solve their problems with using your product.  With your focus on making sure you help your users solve their problems, you may overlook the problem that you introduce – learning how to use your product.  So, you better make sure you understand the nature of how people develop competence and expertise.

Malcolm Gladwell, in Outliers, builds on Dr. Ericsson’s analysis that it takes 10,000 hours of doing something to become an expert at it (not everyone agrees).

Defining Competence

The first step to measuring competency is to define the model.  I am proposing a definition in this article that I suspect will yield insights (to help us manage our products). I was unable to find any quantified definitions of competence, when researching it as part of a client engagement.  If you have, or know of a model, please share it in the discussion below this article.

• A competent user is someone who learns to perform a task in half the time it initially took them.
• An expert user is someone who can complete a task in 10% of the initial time.

This definition is guided by an expectation that Alan Cooper’s premise about perpetually intermediate users is true.  Being a novice user is a very transient state, and becoming an expert is very infrequent.  The goal of the definition is to be able to segment your users and make well-informed design decisions.

Modeling User Competency

Building on these definitions (and the hypothesis that Cooper is correct), you can model the level of difficulty of task-learning that best represents your product (see the article) to get insight into how rapidly your users will evolve their competency over time.  It can impact your modeling of ROI-realization, but more (?) importantly, it gives you insight into understanding how quickly your users will tire of the hold-my-hand-wizards you coded to guide them through your product.  As an example, their (measurable) improvement may look like the following:

[larger image]

Of course people are rarely chained to their desks and using your product continuously.  Even call-center employees get 5 to 8 minute breaks every hour.  So you should develop some insight into how quickly ( in hours / days / weeks) your users develop competence.

The following graph shows how an 80% learning curve overlays a calendar for tasks that happen daily, weekly, and hourly.

[larger image]

The graph shows that for a weekly frequency, after 16 weeks, the task time has reduced from 300 seconds to 100 seconds. With a daily frequency, the task time is even lower – about 60 seconds. This graph shows nothing other than converting the academic learning curve graph into one that incorporates calendar time and frequency of occurrence.

Software Usability and Learning Curves

Part of the catch-22 of developing products is that by trying to be all things to people in all-levels of competence, you may try and just add features.  Over time, this can really add up.  So, you have to figure out the right amount of capability to build into your product.  You can play a couple tricks, whipping out your handy Kano analysis skills again, and prioritizing capabilities and features with their characteristics in mind.

Consider the more is better requirements. Think of them in two categories – user interaction improvements, and application performance improvements.

User interaction improvements remove complexity, and make software easier to use. This results in more user happiness from a given feature, and also allows us to implement more features at a given level of happiness (appeasing salespeople).

Application performance improvements don’t create as dramatic of a shift (they don’t make the application easier to use). They do, make it more enjoyable for a given feature set – shifting the curve up.

Summary

Failing to take into account the learning journey that your target users go through is just a more nuanced version of the elastic user problem.  The easiest way I’ve found – so far – is to treat “learn how to use your product to achieve their (other) goals” as a user goal, and create a distinct persona that represents your target user while they are a novice.

By analogy, one persona for the caterpillar, one for the butterfly.

If you know of a better way to deal with this as a product manager, please chime in below – this solution is a bit more clunky than I would prefer to use.

This article continues the series exploring the root causes of product failure.  Even when you target the right users, and identify which of their problems are important to solve, you may still fail to solve the problems sufficiently.

Why Do Products Fail?

Imagine the one year anniversary of your product launch.  The head of your business calls a meeting and asks, “So, why did the product fail?”  Defensively, your boss explains that it wasn’t product management’s fault – you picked the right users, did the research, and identified the problems that they care the most to solve - and are willing to pay to solve.  And that’s what went into the roadmap, and solutions to those problems are what the team built.  At that moment, the nagging, amorphous thought that had been bugging you finally crystallizes and you chime in.

Maybe we didn’t solve the problems enough.

Enabling Users to Realize Value

This article builds on the framework below (from Why Do Products Fail? – Solving the Wrong Problems)

[larger image]

Specifically looking at the branch of root causes in the lower-left corner.

There are two different ways that a product may fail to sufficiently solve the problems that are important to your users.

• Does not address enough of the user’s needs – teams usually catch this when talking about “table stakes” for a product.
• Does not sufficiently address the needs that are addressed – this is the trickier one, knowing when something is “good enough.”

Unless you have a relationship with your customers built on trust – they trust that you will eventually “get there” – you will have a tough time convincing someone to buy a product that is “not ready for prime time.”  [I hope that that is the last of the trite phrases describing minimally viable products, but we'll see how it flows.]  The difference between the two root-cause branches is that one is top-down (e.g. the sum of solutions is not enough), and the other is bottoms-up (e.g. individual solutions are inadequate).

The Minimum Viable Product (MVP)

It is always fun to talk about the latest shiny object in the product management lexicon.  Talking about MVP is particularly fun because the concept is incredibly valuable, nuanced enough that many non-product (management | marketing) people don’t understand it, and MVP is at high risk of being misused.  Most of the confusion that I’ve seen comes from placing the emphasis on the wrong word.  Which is tough, since there are only three.

1. The Minimum Viable Product
2. The Minimum Viable Product

That’s it in a nutshell.

Most of the best practices of software development, in particular agile and lean* methodology, bias us towards focusing on – almost fixating on – the former.  How quickly can we get the minimum product out the door?  This isn’t a criticism of the “message of agile” – just an assessment of what happens to a lot of teams, as a result of hearing the message.  I suspect, for some people, when you emphasize minimum, the message that is heard actually excludes the under-emphasized word.

When someone says “minimum viable product”, the audience hears “minimum [mumble] product.”

It makes sense that the emphasis has been on the word “minimum.”  Agile’s David needed a potent sling against waterfall’s Goliath.  Iterative development is built on the premise that you can discover, along the way, the best product – and that you can’t, in advance, (completely) determine what the right product will be.  A major component of the value proposition of agile has been that by releasing earlier, you start getting revenue earlier.  Get something valuable into the market now, start delivering value now, start realizing revenue now.  As the product gets better, you’ll get more sales, and accelerate the timeline for realizing revenue.

This message resonates effectively with business leaders who are willing to “give it a try.”  The problem comes in when those leaders forget that “go to market early” still has to be coupled with “finish the product.”  The old mindset of “build until launch, then enter maintenance mode” is the problem.  You can’t have your cake and eat it too – when you release early (and you should), you need to keep developing until the product is done (viable).  Only then do you move into a maintenance mode.

When someone says “minimum viable product”, for some reason, the audience hears all three words.  The reaction is “yeah, viable, but what, precisely, do you mean by viable?”

We’ll come back to that – it’s easier to describe the top down by referring to the equivalent concept from the bottoms up.

And that’s exactly the right question to ask.

*Eric Reis describes the MVP as the minimum required to get value (for you) – which could be simply learning about the market, a precursor to succeeding in the market.  That’s actually a very good thing, and very useful – the only risk is that you forget that a particular release is an experiment designed for learning, and not a product designed for success.  So – don’t stop experimenting, just make sure you don’t confuse the laboratory prototype with the market-viable product.  Make sure you keep investing until you have a product that is both minimum and viable.

Solving the Problem Enough

In addition to solving enough of the problems that a user faces (to make the product viable), you have to solve those problems sufficiently.  This is the bottoms up view.  When describing how effective your product is at solving a problem, there are really only four possibilities.

1. Your product does not address the problem, and therefore does not solve it.
2. Your product inadequately addresses the problem, and therefore does not solve it.
3. Your product adequately addresses the problem, thereby solving it.
4. Your product does something to redefine the market, in such a way that the problem becomes irrelevant, thereby effectively solving it.

When it comes to knowing if you’ve solved a problem enough to satisfy your users, Kano analysis is the right framework to use.  Specifically, you are focusing on the more is better problems.

[larger image]

For these problems, there is a range of possible solutions.  The homepage takes too long to load.  How fast does it need to be?  I spend too much money on gasoline for my car.  How much does fuel efficiency need to be improved to solve this problem for me?

One of the rules of writing requirements is to make the requirements verifiable (and by implication, measurable).

A structurally sound*, well written requirement may look like the following:

• Reduce the elapsed time to provide the estimated price for an average customer request from 7 minutes to 3 minutes or less.

*Assuming that “average customer request” and “elapsed time” are clarified to be unambiguous, and the user doing the providing is also identified.

The question that is the crux of this issue comes from understanding why the requirement-author picked 3 minutes.  Would 4 minutes be inadequate?  Does the goal really need to be 2 minutes?

Perhaps a business case for the product was defined, and moving from 7 minutes to 3 minutes is what is required for the project to comply with the cost-model built into the business case. Most of the teams I’ve seen would have said “3 minutes sounds good – it is a 57% improvement” and patted themselves on the back when they achieved it.  The better teams would have referred to the business case, demonstrated the ROI of achieving the requirement versus the costs of the project, and patted themselves on the back when they achieved it.

Notice that neither of those approaches is based on an analysis of what the user considers to be acceptable.

The best teams would have taken into account user-adoption (or abandonment) rates when building the business case.  Those teams would have immediately raised the question – “How good does it need to be to satisfy the users?”  Then they would have put in the effort to figure out what the user needed.

The minimum viable capability for solving the problem in this example is to provide the estimated price in 2 minutes or less.

Solving Enough of the Problems

For each problem, you need to solve it enough – or you won’t get credit for solving it at all.

Just as people have a notion of “better, but not good enough” that they apply to the continuum of solutions to a particular problem, they have the same notion when applied to a set of problems.

When defining a product, you may identify a user who has five problems that you want to solve with your product.  It may be that users are willing to consider any product that solves 3 of the 5 problems to be good enough – but not a product that only solves two of the problems.

The minimum viable product is one that solves 3 of the 5 problems.

See, much easier to get the idea across after looking at the single-capability view first.

Summary

Even after identifying the right users, and the right goals (problems), you have to solve enough of those problems, and you have to solve them well enough.

Having an outside-in bias as a product manager is important – you need to understand how your customers (or your customer’s customers) would value capabilities you might build into your product.  When running a workshop to collect that information, playing some “serious games” is a great way to get more and better information.  We ran a few 20/20 Vision games last week, to great effect.

Innovation Games

If you are a product manager (or play one on tv), and aren’t already familiar with Innovation Games, you should be.  These are games that intentionally structure different brainstorming and elicitation exercises to be effective at eliciting information from participants.  As a bonus, they are engaging, fun, and easy to run.  They are effective because of the psychological biases that are implicit in the way they are structured, and because they are fun.  I have been shamelessly stealing pieces of what Luke Hohmann (he also runs Enthyosis) and the rest of the Innovation Games team are doing ever since the Innovation Games book came out.*

Product Management Process

When I’m teaching product management, I use a version of the following diagram to describe the product management process

[larger version]

The 20/20 Vision innovation game helps in the first steps of this process – gathering market data, and gathering prioritization data to help with the analysis of the gathered data.

Customer Journey Mapping for Product Management

Customer journey mapping is a tool developed in the user experience domain, designed to get a view of the customer’s experience – taking into account a more comprehensive view than you would get from a storyboard – which would capture a single-purpose set of interactions.  Chris Risdon of Adaptive Path has a great presentation explaining customer journey mapping, from the 2012 IA Summit.  He’s one of the thought leaders to whom you should listen.  Customer journey mapping helps you create a framework for comprehensively understanding and improving how customers will experience your product in the context of their reality – and from the customer’s point of view.  It is a great tool for anticipating the emotions that customers (users) will experience as they interact with your product(s).

By analogy, if a wire frame is a static view of the product with which someone interacts, a journey map is a dynamic view of the experience they will have.  As a product manager, I saw a great opportunity to co-opt (much of) the journey mapping approach as a way to organize and understand the needs of the users, for a complex set of B2B2C products.

Understanding the B2B2C (Business to Business to Consumer) Model

For folks who don’t know B2B2C, it is “business to business to consumer.”  If your company is an OEM (or the software equivalent), and your customers are other businesses, who use your products to service their customers, you have a B2B2C product.  This extra layer of abstraction makes it more challenging to be develop Outside-In software when figuring out what is most important to the end users.  As a B2B2C product manager, you need to identify not only the things that are important to your (direct) customers, but also the things that are important to their customers.

[larger image]

The above diagram shows a simple example of a B2B2C relationship – sometimes pictures work better than paragraphs.

Through the course of a two day workshop with a couple dozen end-users, we identified a few hundred interactions / touch-points / experiences that end-users have throughout a customer journey that was framed in 26 steps.  Anecdotally, some steps would look exactly the same to the “middle (B) customer”, but appreciably different to the end (C) customer.

One insight that we wanted to capture was the relative importance of each step in the journey to the (end) customer’s perceptions of the experience.  This would allow us to inform our hypotheses about the relative importance – to the middle (B) customer – of investments in improving the end (C) customer experience.

This prioritization exercise – identifying which steps were most important to users (end customers) – happened after identifying the steps in the journey and the important interactions within those steps.

We had teams of 4 to 6 people, each representing different personas (because you should not design a solution for “everyone”, but rather for groups of people) working together and in parallel throughout the stages of the workshop.  Each persona-group was then asked to play the 20/20 vision game, to put the 26 steps in relative priority order.  The exercise proved to be pretty quick and engaging for each team.  We gathered some good qualitative insights from the debates within each team, and from the analysis of similarities and significant differences in relative-importance across groups.

We also had the teams, for those steps that were identified as the most important to them, play the 20/20 game again.  This time, the teams put the individual interactions / touch-points within each step into relative order by importance of the interaction to their perception of customer experience.  This time, the games were played with more like a dozen items to prioritize at a time.  Some steps had more than 50 interactions identified, but with duplication (we did exercises to cluster equivalent interactions together), the number of items being put in order was consistently below 20.

Conclusion

Being market-driven is important when defining which problems products should address (or address more effectively).  The right place to start is by internalizing the customer’s perspective on the actual problems to solve and the context in which they live.  A precursor to doing this is identifying who the customers are.

The next step is to form hypotheses about the value – to the customers – of solving these problems.  Co-opting the mechanics of customer journey mapping, and adding in a couple prioritization exercises worked well for me, in achieving these goals.

Next up – developing insights from what this effective data-gathering exercise yielded.

Attributions & Disclaimers

Thanks Michael Cory for the eye chart image

*I know Luke personally, have done agile product management and training work for/through Enthyosis in the past (and hopefully will again in the future).  I’m a big fan.  So yes, this article reads with a bias – but only because the stuff is both good and useful to me and I believe it would be for you.

Why Do Products Fail?

This series is following a root-cause-analysis approach to understanding reasons that a product will fail in the market.  A slightly different approach of “remembering the future” is being used.  Imagine that your product fails in the market, and that you’ve been tasked with figuring out why it failed.  You start by categorizing the reasons it may have failed, then start drilling down into the reasons behind the reasons.  That analysis will lead you down many paths.  To follow the series from the top level to this point, if you haven’t been reading along already, please review the following articles:

• Why Do Products Fail? (Reasons a product fails in the market, including not solving the right problems)
• Why Do Products Fail? – Solving the Wrong Problems (Reasons a product does not solve the right problems, including not enabling users to realize value)
• Why Do Products Fail? –  Picking the Wrong Users (Exploring one branch of the reasons that users do not realize value)

The specific reasons roll up to the following categories (read it as “The product …” :

• … Does not target the right users - you may have solved someone’s problems, but not the right someone.
• … Does not focus on the important goals of target user personas – you may have picked the right users, but you failed to focus on the most important goals that they have.
• … Insufficiently addresses the user persona’s needs - even with your eye on the ball – the right problems for the right people – you may not solve those problems “enough.”
• … Does not account for user persona’s level of experience - as people use a product, the nature of the problems they face evolves – did you take that into account?
• … Does not incorporate the context of usage – solving the right problems, sufficiently, for the right people is not enough – you have to account for the context in which they care.

When looking at the details under each of these categories, we get to the root causes of failing to enable your users to realize value from your product.

[larger image]
Why Do Products Fail? – Picking the Wrong Users

Where that article explored the Does not target the right users branch, this article explores the Does not focus on the important goals of target user personas branch.

Important Goals

Assuming you’ve already identified the right users (a focus on satisfying the preconceptions of the right buyers would lie in a different branch), you need to then have an understanding of what is important to them.  Failing to focus on the important goals will happen if you (1) fail to identify the user’s goals, or (2) fail to appreciate the relative importance of each goal the user has.  Failing to identify the goals is obvious – if you don’t discover the goal, you won’t solve it (except accidentally – you could get lucky).  The other aspect of failing to identify important goals is not recognizing which of the identified goals are actually important to solve, for your target users to realize value.

As a side note – solving (important) problems in the wrong order, solving too few problems, and providing insufficient (partial) solutions to problems are all captured in other branches of the Ishikawa – this branch is solely about failing to identify important user goals and failing to appreciate the importance of those identified goals.

The first step comes in understanding your users.

Elicitation

You don’t want your users to define your product – you want them to inspire it.

Henry Ford’s apocryphal “faster horse” quote comes to mind of course – we’ll never be free of it (see Patrick Vlaskovits’ article on the HBR blog – a fascinating read regardless, including Alfred Sloan’s market segmentation approach – “A Car for Every Purse and Purpose“).

The goal of elicitation is not to be told what to build, but to develop an understanding of why something needs to be built.  The last question you want to ask is “what do you want?”  You need to start by trying to get users (or potential users) to tell you about the problems that they care about solving.  In my experience (excluding other product managers and business analysts), when you ask someone to articulate the problems that they face, they will unerringly tell you about problem manifestations instead.

If you ask a user to tell you what is wrong with an eCommerce site, he will tell you that the search is too slow, or that the results are not relevant.  Those are problem manifestations.  The problem the user is facing is that he cannot find what he is looking for.

When I’m describing the process of product management, I use the following diagram:

[larger image]

Asking questions is a form of market research, and the answers are data.  You can’t jump directly from those answers to a roadmap or backlog – you have to analyze the answers to generate insights.  In the case if elicitation, you’re developing insights about what the underlying problems are.  It is the analysis and synthesis that leads you to discovering the underlying problems.

Tom Chi describes it as framing the problem correctly.

Roger Cauvin gives us some great advice on interviewing prospects that warns us from making some common mistakes.  The easiest way to get past problem manifestations and discover underlying problems is to ask “why?” and then ask it again. Roger also gives us advice on when to stop asking why – when you reach the core.  You can visualize this with Ishikawa diagrams (much as this article series is attempting to do).  Using this article series as a meta-example, picking the wrong user goals is a problem manifestation, product failure is the problem.  Roger (again) provides some great insights in an analysis of an article by Karl Wiegers, where he provides some good real-world examples of identifying user-needs.  Another handy trick for elicitation – ask provoking questions to stimulate lateral thinking.  While we’re piling up the articles for deep-diving, here are some key active-listening techniques that you can use not only to assure that you understood what someone is saying, but to probe into the root-causes and discover the underlying problems.

Prioritization

At the end of the day, you want to prioritize what you build in terms of maximizing your bang-for-the-buck.  This has proven to be such a valuable framework that the Innovation Games folks have even created an online game out of the bang-for-the-buck technique.  Given your available resources, it lets you maximize the rate at which you deliver value per unit of time (in the article, the focus is on prioritization within and across sprints, but conceptually it applies regardless of your development cadence).

Assuming you’ve picked the right users, and identified the problems that they care about solving, the key is to solve them in the right order.  You may find you’re in a unique circumstance, but start with the point of view that “most valuable to the user” is the measure by which you should assign importance to the user.  It is also imperative that you are thinking in terms of differentiated value relative to alternatives.  Some abstractly-valuable problems already have solutions that are readily available to your users.  Use the point of view that not-yet-solved problems are where the value is.

One framework for understanding the importance of (solutions to) problems is Kano Analysis – which you can use to identify if users view solutions to problems as more-is-better, delighting, or must-have.

There are actually several games from Luke Hohmann and the InnovationGames people that can help with this – here are a few:

• 20/20 Vision – get customers to put solutions to the problems into relative order (for them).  Effectively, a card-sorting exercise.
• Speedboat – the relative priority insight you get from this game comes when people put anchors at different depths, indicating the severity of a particular problem.
• Whole Product – this brainstorming game can also be used to identify what people perceive to be Must Be (table-stakes) capabilities of your product.

Once you identify the importance of solving particular problems to your target customers, you have half the information you need to sequence what you will build (you also need cost estimates for creating solutions to those problems, to use bang-for-the-buck).

When you’re doing an analysis of the importance of problems, you’re usually looking at multiple groups of users.  In the Important Problems article in the series on comparing products, you can see a method for visualizing the importance of multiple problems, to multiple users:

[larger image]

The above table shows hypothetical data representing the “quantified” relative importance of having a solution for each problem, to each persona / context.
Important Problems – Comparing Products Part 4

Summary

Even with an understanding of who the right users are, you still have to understand which problems are important to them to solve, and in which order you should solve those problems.  That order should map to maximizing the value you can provide them in the minimum amount of time – achieving bang-for-the-buck, where your users define the bang, and you control the bucks.

Exploring the reasons that a product might fail in the market is a useful way to triage and assess what you need to do to prevent the failure of your product.  Instead of taking the “do these things” approach as a prescriptive recipe for product managers, I’m approaching the exact same topic from the opposite direction.  I was inspired in part to explore this approach when thinking about the Remember the Future innovation game.  Instead of asking “What will the system have done?” in order to gain insights what it could be built to do, I’m asking “Why did your product fail?” in order to prevent the most likely causes of failure.

Exploring Causes of Product Failure

In the first article on why products fail, I started with the top-down categorization of sources of product failure.

[larger version]

Within that framework, we started diving in to the causes of product failure that could be broadly described as “[the product] does not solve the right problems.”

[larger version]

Breaking this down involves exploring why the problems are not the “right” problems to solve.

[larger image]

One branch of the Ishikawa involves failing to solve the problems that are important to the user personas.  That’s what we’ll explore in this article.

[larger image]

There are several ways in which you could reach the conclusion that user personas have failed to realize value.

[larger image]

The specific reasons roll up to the following categories (read it as “The product …” :

• … Does not target the right users - you may have solved someone’s problems, but not the right someone.
• … Does not focus on the important goals of target user personas – you may have picked the right users, but you failed to focus on the most important goals that they have.
• … Insufficiently addresses the user persona’s needs - even with your eye on the ball – the right problems for the right people – you may not solve those problems “enough.”
• … Does not account for user persona’s level of experience - as people use a product, the nature of the problems they face evolves – did you take that into account?
• … Does not incorporate the context of usage – solving the right problems, sufficiently, for the right people is not enough – you have to account for the context in which they care.

When looking at the details under each of these categories, we get to the root causes of failing to enable your users to realize value from your product.

[larger image]

Focusing on the Right User Personas

Developing a set of user personas to guide the development of your product is important for several reasons.  It provides a framework for focus – only building what is needed by the people who will actually use the product, not building what someone thinks “would be cool.”  It provides a framework for prioritization – yes, some users will do that, but most won’t, so we’ll do that later.  It provides context for the team to avoid a common manifestation of not having defined users – the elastic user problem. It gives you an opportunity to build a product that people love – creating loyal customers.

The first challenge is to make sure you identify the right users. The approach I describe here may only make sense in the context of the customer-centric market model that I use for thinking about markets – I’m not sure, it is so wired into my approach now that I may fail to clarify why I make some statements (if so, go read the earlier article).  In a nutshell, this approach defines a market as follows:

Markets are made up of market segments, which represent groups of customers who share common problems that people will pay to solve.  The notion of “customer” is combining both buyer personas and user personas to simplify the model.

Many products are relatively straightforward – within a given market segment, there is only one important user persona.  For many B2B and enterprise software products, there are multiple user personas who collaborate to solve business problems.  You need to understand the ecosystem of how your customer operates in order to identify the user personas that are important.  Sometimes, the important user personas are not the people who directly interact with the system – they can be people who “use” the outputs of the system.  In an article on visualizing stakeholder analysis, I show how to use an onion diagram.

[larger image]

In a nutshell, there are direct users of the system, other employees (of your B2B customer) who are part of the containing system, and still other actively involved “users” in the wider environment.  You identify these “non-user users” by understanding the interactions and goals of the [traditional] users.  You could choose to focus only on the actual users, or identify the “other” users that make up part of the ecosystem.  Both approaches work.  The same user story would be syntactically different:

As the pest control company admin, I need to be able to quickly see the history of our relationship with a customer when she calls me, so that I can deliver personalized service, so that I can improve our customer’s satisfaction with our service.

As a pest control company customer, I want to know that my company cares about me, because I’m willing to pay a premium to support local merchants as long as I get personalized service.

I suspect it is a matter of personal (team) preference, but I prefer treating the non-user users as first class citizens.  I believe you are more likely to generate insights about the problems the user-users should care about solving, when you start getting inside the heads of their customers – a key approach to developing B2B2C products.

Once you’ve identified the users, you need to identify the relative importance of each group of users.  Choosing who to satisfy – or who to satisfy first is a manifestation of your go-to-market strategy.  Selection of the “right” strategy is a little too broad of a topic to cover here, a thousand words into an article on user persona prioritization.  As part of a series on comparing products, I made the case that you have to compare products from the perspective of users, and that the comparison should be weighted based on the relative importance of the users.  In that article, I touch on market size, cross-market influences, higher-level strategy, risk mitigation, and ease of entry (into the market) as key areas of focus in strategy formulation.

The methodology for weighting the comparison in the context of user goals and the relative importance of users can also be used to prioritize investment decisions in your product.

Summary

As the detailed Ishikawa implies, there is more to come about failing to address the needs of your users.  More to come in future articles.  To summarize this installment,

• Your product failed because you failed to solve the right problems.
• You failed to solve the right problems because you did not enable the users to realize value.
• You failed to enable users to realize value because you either (a) failed to identify your users, or (b) failed to prioritize solving problems for the right users.

To avoid product failures that stem from these root causes, identify your users, and their relative importance in the context of your particular strategy.

There are many reasons that a product might fail in the market.  One of those reasons is that your product solves the wrong problems.  There are many ways to solve the wrong problems.  This article continues the series on sources of product failure, exploring the idea that your product may be trying to solve the wrong problems.

Quick Recap of Product Failure

In an earlier article on why products fail, I framed the problem as a root-cause analysis, using an Ishikawa diagram to categorize the sources of product failure (in the market).

[larger version]

There’s a great discussion in the comments on the first article – focused mostly on “picked the wrong market” and interpretations of positioning.  The initial article frames the reasons for product failure as being discoverable by exploring one or more of the following branches of the Ishikawa:

• Business Case is Flawed
• Picked the Wrong Market
• Takes Too Long to Enter Market
• Doesn’t Solve the Right Problems – the focus of this article
• Positioning & Sales Approach is Wrong
• Product is Not Good Enough

In this article, the focus is on solving the right problems.

[larger version]

Solving the Right Problems

How do you know if your product is solving the right problems?  There are two questions – did you pick the right problems to try to solve, and did you succeed in solving those problems?  The focus of this article is entirely on the selection of the right problems, not on how effective your solution is at solving them.  The “Product is not good enough” branch would capture scenarios where you picked the right problem, but then your solution failed to deliver a valuable solution.

[larger image]

Broadly classified, there are 6 reasons that your product may not be solving the right problems.  All of these branches represent a failure to make the right decisions based on external factors – being market driven, or outside-in in your perspective about what needs to be done.

• Doesn’t enable user personas to realize value – the focus of this article
• Doesn’t address buyer persona’s perception of problems
• Doesn’t align with (customer) company’s strategy for competing in their market
• Is not differentiated from alternative solutions
• Did not solve the problems in the right order
• Did not solve enough problems – or solve the problems enough

Enabling User Personas to Realize Value

One aspect of being market-driven is being user-centric.  To do that, you have to identify who your user personas are, and what they care about.  Users have goals, and user personas are archetypes designed to help you create products for groups of users that share common goals.

When you’re developing consumer products, creating user personas is sufficient for describing groups of users.  When you’re creating B2B products, you’re trying to address the needs of people that operate within an organization – and often there is a lot of complexity within that organization.  You can organize your insights into which people within your customer (company) perform which roles using actor hierarchies – a common tool for mapping use cases to particular roles.

For example, a call center manager will make sure calls don’t wait in the queue for too long, while a call center employee will make sure that each call is handled efficiently.  However, not all call center employees are the same either.  Often there are multiple roles, or tiers, of call center employees – designed to reflect levels of expertise.  Since most calls are straightforward, newer or less experienced employees are often tasked with handling the incoming calls, and then redirecting them to more experienced or specialized experts only when needed.  This allows a company to more efficiently utilize the time of the (more expensive) experts.  All of these employees will share some goals and tasks, but the more specialized employees will have additional tasks and goals.

You may also use hierarchies to reflect variations in roles – see this article on using actor hierarchies to address regional differences in a global business.

It isn’t sufficient to identify who your users are, you have to also identify what they care about.  The combination of “for whom” and “why they care” sums it up.  In an earlier series on comparing products, my point of view builds around the idea that your framework for comparison – if you want to be honest – should come from your customers.  That approach and series has a lot of overlap with this article.  After identifying the personas for your product, the next step was identifying the problems that are important to them to solve.

From that series – a view of the problems that Tim, a niche-content consumer, cares about when considering a mobile content consumption device:

[larger image]

If you don’t understand that Tim is your customer, and you don’t understand what Tim cares about, the only way that your product will satisfy Tim’s goals is through a happy accident.  If you aren’t being intentional, then definitely explore this branch as a reason why your product may have failed.

Summary

Just setting the framework for figuring out if you’re solving the wrong problems for the wrong users took a little more writing than I expected.  The next article in this series will continue exploring understanding if you’ve got the right customers in mind.

In the first article of this series, you gave some fantastic insights and poked at the framework very effectively.  Does anything jump out at you as missing, errant, or broken with this branch of the root-cause analysis?  Chime in below!

Are product managers really involved in strategic discussions, or are we just order takers?  Adrienne Tan has poked the beehive and started a great discussion with this article.  Joining in from here, hopefully adding folks to the conversation.  Check it out, and chime in here or on the brainmates blog.

Product Managers Taking Orders

Adrienne kicks off the discussion with a great post, including the following question : “why does a whole professional group continue to defend its right to be strategic? No one else seems to think that Product Management is the rightful owner of Product Strategy except Product Management.”

As I write this, there are half a dozen great comments on her post, including some powerful ideas:

• People don’t want to relinquish power – and “owning strategy” is powerful.  Of course other people want to say that they “own” it.
• Product management is the business – we run into problems when the role is “tacked on” organizationally and not deeply integrated.
• There are two distinct roles – strategic planning and tactical support – and both have the same title (product manager), but if they are different people, you have problems.
• There’s too much for one person to do, but the responsibilities of people sharing the work must overlap, or they will become disconnected.

As Nick points out in the comments – Adrienne’s post is a productive one, not just a rant – since getting a “seat at the table” for strategic decision making is so hard, is it worth doing?

Product Strategy

Product strategy happens.  It may be implicit, but it is probably explicit and intentional.  Product strategy, however, is just a business tactic.  Your company has a strategy, and someone makes the decision that “product” will play a role in that strategy.  The definition of that role constrains what someone else should do with the product, in order to realize the product’s portion of the business strategy.

Most of the product management roles that I’ve seen fall into this model.  There’s a “glass ceiling” for product managers – who are only given freedom to make decisions within this context.  Those product managers are “doers” within these constraints – occasionally allowed to, but generally not encouraged to, and certainly not required to provide recommendations to change the business strategy.

Go Where I Tell You

If the CEO (the actual CEO, not the “CEO of the product”) is the rider, then the product manager is the horse – constrained to “go where I tell you.”  The product manager is watching where he’s running to make sure he steps sure-footedly, looking around to see where the other horses (or wolves) are – basically responsible for the “running.”  The CEO is responsible for knowing where to go, not how to get there.  As long as the product manager doesn’t get the bit in his teeth, the CEO can make sure the horse goes where she wants.

What Does “Strategy” Mean to You?

The problem with words like strategy is that they carry a lot of symbolic baggage.  Wikipedia tells us that a strategy is a “plan of action, designed to achieve a vision.”  The question is one of scope – what level of “vision” are you trying to achieve?  Google’s vision is so big that they don’t really articulate it – instead, they share the philosophy that shapes their vision and guides their actions.  You have to pick one of the products, like Google Wallet, before you get an articulation of vision.

Product management (as a “named entity” in the organization), in the teams that I’ve been a part of, has “owned” the definition of the product strategy that enables the product vision – but not been invited to participate in the business strategy that enables the company’s vision.  That strategy is primarily embodied in a product roadmap that articulates how (and when) the product vision will be achieved.

The product strategy and vision are components of a portfolio of strategies and visions that collectively make up the business strategy.

Command and Control

The weakness of this old-school command-and-control model is that the commander (CEO) only gets limited inputs from the commanders in the field (product managers).  With this top-down model of decomposition of business strategy into products with specific roles, each product manager has specific responsibilities (take the town), often being expected to succeed with limited context (we’re sweeping around the enemy’s flank, but it is a feint).  The product manager is only providing progress reports and possibly market information back to the overall commander.  The product manager is not tasked with providing recommendations to change the business strategy.

The strength of this approach is that it enables a company to execute their business strategy.

As product managers, we know there is always “more to be done” – and like the metaphor of boiling the ocean, if one person tries to do everything, they won’t succeed.

In his recent series on product management roles, Rich Mironov calls out that the VP of Product Management is “making sure that the company as a whole is building and shipping and supporting the right products.” Not defining the business strategy, but rather determining which products should be components of that strategy, and what roles those products should have as members of that strategy.

Each product manager is responsible for defining the roadmap for their product, articulating how it will fulfill its role in the comprehensive strategy.

The Eye of the Beholder

Given this definition, is product management still a “strategic” role?  It depends on where you’re standing in the organization.  The CEO probably sees us as horses.  But the teams that are building, testing, and supporting our products are operating on a shorter time horizon, with an even narrower scope.  From that point of view, having a multi-release point of view about the product, and a deep understanding of the market (needed to make the right product decisions) is definitely strategic.

Should We Be Pushing the Strategic Product Manager Agenda?

Getting back to Adrienne’s original point, and a fantasic image from Geoffrey Anderson, “Sometimes I feel like a salmon swimming upstream to spawn, and every tier of the swim is fraught with bears waiting to eat me.” – is it worth investing our time and energy in pushing the “product managers are strategic” message?

If any of these apply to you, then yes – push:

• I’m told what features to build, and I’m expected to just translate for the technical folks on the product team, so that they build it correctly.
• I’m not given the (business strategy) context that defines the role my product plays, so have no way to know if my product is succeeding.
• When I provide feedback about the feasibility or effectiveness of my product in its role as part of the business strategy, it is not valued and nothing changes.

If none of those apply to you, they do apply to some of your peers – so push.

In the past 10 years, I’ve seen each of the above situations multiple times, and they are in order of increasing frequency.  What I see even more frequently these days is product managers getting pulled more and more into the operational, product owner role – shepherding teams through daily stand-ups and validating acceptance criteria – purely tactical execution roles.  Those product managers are still somehow responsible for doing product management, even when not given enough time to do it.  If that’s the position you find yourself in today, start winding the klaxon.

Attributions

Thanks capl@washjeff.edu for the bridle image.

Thanks Bryan for the wargame image.

Why do products fail?  Trying to organize all of the reasons that your product might fail is a Herculean effort.  Understanding how your product did, will, or might fail will help you focus on what you need to do next.

An Exploration of Product Management

A personal goal for me is to become better at product management, so that I can help create better products.

As a product manager, the most important thing you should be doing is understanding the problems that your customers face.  If you treat “improving at product management” as the product, you should start with an exploration of the problem space.  I’m framing that problem space as products that fail.  I think it is also fair to also think about products that under-perform.  They “succeed” given the goals by which they are being measured, but they never realize their full potential.  I’ll keep the “not as good as it could be” notion in the back of my head, and talk to “failed” as the larger issue.

There are conversations, blogs, books, processes, and frameworks for “how to be a (good) product manager.”  I suspect looking at this from the outside-in (problem first, solution second) may yield some interesting insights.  Thanks to Leisa Reichelt for her article on why most UX is bad, which inspired me to have the conversation here, approaching the problem as a root cause analysis.

As an agile product manager, I’m going to approach this iteratively.  I hope that you will provide insights and corrections, helping to adapt this as we go.  Your contributions will make this better, faster.

Root Cause Analysis of Failed Products

Ishikawa diagrams were originally created to allow engineers to figure out why something broke – a visual tool for organizing root cause analyses.  I’ve been using this powerful decomposition tool for several years to solve problems and organize my thoughts.  It may provide the perfect framework for gaining understanding about why products fail.

Let’s dive right in.  Here’s my first crack at the very top level of an Ishikawa for product failure.

[larger version]

Looks pretty sparse (for now).  I will fill it in, as I drill down into each area.

If you’re like me, you’ve got some “reasons for failure” in your head right now – maybe from past experience, maybe from watching products in the market today.  Do any of those reasons not map into the categories above? Tell us about it in the comments (or tell me privately, if you must) – that’s the first vector for informing an improved version of this diagram.

Here are some prose descriptions of what I’m thinking about (for now) for each of those branches on the diagram – I expect them to fill out with smaller branches attached to each of the main branches.

Product Fails in the Market

• Business Case is Flawed – This is where we would capture things like a product strategy that is not profitable, for example, your model was dependent on exponential growth – so even though you had consistently growing market share, it wasn’t enough for the product to be considered “a success” by you.
• Picked the Wrong Market – Maybe this market is about to go away, like buggy whips or audio cassettes.  Maybe the competitors in this market are just too good.  Maybe entering this market is too divergent from your corporate strategy and dilutes focus and investment in your company’s other products.  Another example would be if your team does not have the skills and resources needed to win in a particular market.
• Takes Too Long to Enter Market – Whatever it is you’re doing to enter the market, it took you too long.  Competitors have “out-gunned” you, your customer’s needs have changed, etc.  This is to capture causes where even if everything else was good, you simply didn’t move quickly enough.  At first blush, I expect organizational problems (lack of alignment, bureaucracy, insufficient resources) will all land here.
• Doesn’t Solve the Right Problems – This is where most product managers focus most of their time – making sure we’re actually solving the right problems (the ones customers are willing to pay to solve).  Problems of design – where we intended to solve the right problem, but the proposed solution doesn’t cut it – would not be in this branch – they would be in the “not good enough” branch.
• Positioning & Sales Approach is Wrong – For the first iteration, this is my catch-all for marketing and sales.  All of the problems that are “your potential customers don’t think of your product as a solution to their problems (even though it is).”  Also the problems of “your potential customers decided not to purchase (when they should have)” and “your potential customers have never heard of your product.”  This is definitely an area where you can contribute more than I can to the framework.  How would you (product marketing managers, I’m looking at you) frame this?
• Product is Not Good Enough – Execution is key here.  Not solving problems completely (although that might move to the “right problems” branch) is an example.  Bad design is an example – both bad user-experience and bad architecture.  Poor execution also lands here – broken windows, sloppy implementations, poor quality.  For this branch to work, “product” is not just the product that your development team builds, but also your customer relationships, distribution, services, etc.

Open Questions in This Model

There are definitely some design decisions I made in the approach above that are worth questioning.  Here are some that come to mind for me – add your own in the comments.

• Designs that fail to solve the target problems – I put this in the the “not good enough” bucket, because I felt like there was value in grouping the “how” separate from the “why.”  Many times, an inadequate design is a design that works great for inadequate requirements – which means the problem is in the “why” column.  How would you organize “bad requirements execution” and “bad design execution” in this model?
• Pricing and Cost – together, they reflect profitability.  ”Not profitable” implies a business case problem.  Incorrect pricing implies a positioning problem or is a red herring that is masking a sales problem.  High cost is reflective of bad design decisions and/or execution problems (both of which are in the “not good enough”) bucket.  Perhaps if you can’t create the product at the cost you need to, for your business model to work, when selling at a given price in order to compete, you have picked the wrong market.  Where would you put pricing and cost issues?
• Team Capabilities and Support. Not having (enough of) the right skills on a team limits what solutions you can create.  Not having enough support to gain needed skills constrains the solution space as well.  When your team does not have what it takes, or have what they need, to create solutions that will succeed in a given market, where would you put this?  For now, it is in the “picked the wrong market” bucket, because trying to compete in that market is infeasible.  There’s probably a better way to frame this – how would you do it?

A Litmus Test

One experiment to validate this model is to look at the main causes of project failure and see if they map well into this space.  In the Chaos Report findings from the Standish Group, the following are the top ten responses from the companies they surveyed, along with my thoughts about how they map into this framework.

1. Lack of User Input – In my experience, this manifests both as not solving the right problems and as delivering a product that is not good enough.  The mechanisms are different flavors of “not listening.”
2. Incomplete Requirements & Specifications – Ends up causing delays, and possibly not solving the right problems, or having designs that are not good enough (because the requirements that drove the designs were not good enough).
3. Changing Requirements & Specifications – I put this squarely in the “your mindset is broken” bucket.  Requirements change, even if your requirements document does not.  Depending on how this plays out (for your team, for your process), you either slog on and deliver solutions to the wrong problems, or you take too long to deliver.
4. Lack of Executive Support – This can be a takes too long problem.  Or it could be that lack of support causes a product to be abandoned (even if it were succeeding), or constrains options for your team to the point that you aren’t able to succeed within the parameters.  This one definitely doesn’t fit the model.  Should we update the model, or treat this one as “out of scope?”
5. Technology Incompetence – insufficient skill results in not good enough and usually delivery takes too long.  If really bad, it results in “impossible (for this team) to deliver.”  For the really bad cases, does it land in picked the wrong market, because that market is infeasible?
6. Lack of Resources – Just a subset of lack of executive support.
7. Unrealistic Expectations – in the worst cases, it is a problem with the business model.
8. Unclear Objectives – lands in not solving the right problem.
9. Unrealistic Time Frames – same as unrealistic expectations.
10. New Technology – yes, change is hard.

Not surprisingly, the list from the Chaos report uses a project-centric language.  Other than the (very valid) “failure profile” that lack of executive support causes projects to fail, the model seems to hold up reasonably well.

Your turn – what would you add or change?

The ideal agile team is made up of specializing generalists – but what does that really mean?  The goal isn’t to prevent functional silos of expertise, it is to allow people to cover for each other.

Great Conversation

Elena Yatzeck (@eyatzeck) posted a comment on an earlier article about agile maturity models.

In terms of refinement, I’m thinking a lot these days about “staffing the engineering team correctly.” I’m not sure I agree in practice that you can or should try to staff all teams with “specializing generalists,” or at least not as taken to an extreme. (If you’ll forgive the self-promotion, I talked more about this here: http://pagilista.blogspot.com/2012/01/no-blender-zone-cross-functional-doesnt.html.)

I’ll not only “forgive” the promotion, I’ll re-promote it.  Good stuff.

When re-reading the maturity-model article, this snippet popped out at me:

People over process is the right emphasis. If you can’t find people that are “good enough” you might as well go home. Doesn’t matter how agile you are if you don’t have the horsepower. You also need people who are excited to “do agile” – they like to communicate, they enjoy the project and team dynamics of an agile process. You’re also better off with specializing generalists – ideally, every member of the team can do any work that is needed. This is an efficiency play – you risk introducing bottlenecks when you have a specialist who is the “only one” who can do particular types of work – because you will not have a consistent mix of types of work from release to release.
Agile Maturity Model

Thirty months later, my experiences have increased my conviction that this is true – and have realized that the way I wrote the quote above fails to provide a key clarification.

Following that link to an (even earlier) article on specializing generalists, brings the following (emphasis added):

The idea of specializing generalists is easiest to grasp by first saying what it is not. It is not staffing a team with a database expert, a user interface coder, a SOA (service oriented architecture) guru and an architect. With four specialists, each development task has an obvious owner. Database changes and refactoring go to the database expert. Reworking the UI goes into the queue for the AJAX hotshot. The problem is that this approach is only efficient when each team member is equally loaded with work. Since an agile team is continuously reprioritizing their work based on repeated feedback cycles as part of each release, this doesn’t work. The team will never face a situation where the (for example) four most important things to do are one item for each specialist. You can very easily have a release where all of the most important tasks are focused on the user interface. So all of the non-interface-experts are either working on lower-priority tasks, or even worse – they are idle. And you delay the most important work until the specialist can get to it.

By staffing a team with people who have an area of expertise, but can do anything, you can maximize the value of each delivery cycle. In our example, where all of the tasks for a release are UI tasks, they can be interchangeably assigned to any of the developers. The UI expert may suggest an implementation approach, do code reviews, or provide guidance to all the other developers. But every developer (including the database guy) can sling code effectively to get the job done. Specializing generalists.

This is very effective for making the “development engine” a black-box. Feed it the highest priority stuff, and it all gets done. We can take that approach to the next level. Designers can implement, project managers can design test plans, and yes, product managers can specify design. Twitch. Back up a sentence and read it again.

Specifying design is not the job of the product manager. True. Very true. Emphatically true. But specifying design can be what a specializing generalist does, even when that person is also responsible for defining market needs.
Specializing Generalists 2008

Elena’s article identifies a common misconception – that “specializing generalist” is a fancy way of saying “a bunch of people who can all do everything:

It’s a seductively simple fallacy of division to interpret the concept of “cross functional” team to mean a “collection of cross-functional individuals.” New agilists are quick to apologize that “we still have functional silos here” as though it would be much better if everyone could do all the same things. Grab some equally skilled poly-functional people, have them all take turns doing all of the jobs as needed, and you’ll all laugh your way to on-time, high-quality, and valuable working software.

Not so fast!

The power of an effective agile team, like the power of any other effective team, doesn’t come from its homogeneity, but from its ability to harness its diversity.
No Blender Zone: Cross Functional Doesn’t Mean Homogenous

Elena goes on to say (emphasis mine)

Team members shouldn’t attempt to Harrison Bergeron themselves into a mish-mash of mediocre (but working!) software. Someone needs to facilitate the stakeholders into some sensible semblance of a business case. Someone needs to build functional test suites that mercilessly beat on the code to prevent it from breaking in production. Neither of these are exactly the same skills it takes to gradually evolve the design of a complex system in modules of 100 lines of code or less. If people want to try new things, that’s great, but it needs to be with the realization that other jobs on the team are actual professions with skills and the need for experience in order to excel.

I completely agree.

Specializing Generalist

Specializing Generalist.

• Not a specialist.
• Not a generalist.

You need best of breed team members who specialize in areas of experise – “actual professions with skills,” as Elena puts it.  Without people who excel in the needed areas, you end up with a mediocre product.  How many times have you gone to the store and asked for the “middle of the pack” product?

That’s not even table stakes anymore.  Just the ability to create “something” isn’t interesting in the market, and isn’t interesting to the members of the team.  How many times have you heard someone brag “I love my job, I’m a cog in the machine?”  You have to have people who specialize in all of the needed areas (interface design, market insight, coding, quality, etc) in order to create a viable product.

If you staff your team with (only) generalists you will fail.

Pure generalists cannot create a product that is “good enough” – because they aren’t good enough at the creating the parts, from which the product is the sum.  You have to have people who specialize in creating great “parts” of the solution.  That’s what you need to have a shot at creating a great product.  But it isn’t really enough.  The problem is in how you define “great.”  Great means that customers buy it, users love it, and your competition is knocked back on their heels by it. Everyone agrees on this, but most people miss one thing.

Your market is changing – you also have to be fast.  You can’t solve the right problems if you aren’t fast, because the problems that are “right” are constantly changing – your market is a moving target.

Specialists, as individuals, are capable of creating great “parts” in their silos, and those parts all add up to a “great” product, so what’s the problem?  The problem is that collectively, by the time the specialists are done, they are no longer solving the right problem.

If you staff your team with (pure) specialists you will fail.

The most important tasks for the team, in any given sprint, will not balance into a perfectly allocated workload, where each “part” is worked on by each specialist, where no one is idle, and no one is a bottleneck.  It just doesn’t happen.  I haven’t seen it in 15 years in the software world, or in my prior decade as a mechanical engineer.

When one specialist is waiting for something important, she isn’t idle, she’s just working on something that is by definition not as important.  OK, you’re minimizing the damage – but you’re still taking damage.  When another specialist is the bottleneck, you lose.  Nothing magical to do here.

If you staff your team with specializing generalists you may succeed.

The work that piles up in any one specialized silo is of varying degrees of complexity.  The “UI specialist” may be backed up with a bunch of CSS tweaks, some straightforward AJAX calls to write, and a gnarly refactoring of the model-view-controller model to adapt to changing understanding of market needs.  No one can solve the MVC problem without specialized skills – but with guidance from the UI expert, one of  the other team members can handle the AJAX calls and CSS updates.  Extend this same model across other aspects of the product.  Your database expert may be needed to optimize query performance or resolve locking problems, but other members of the team could make straightforward schema changes.

It is the collective ability of the team to optimize what they collectively work on that accelerates the team’s delivery of the most important capabilities.

You have to have people who specialize, in order to optimize individual performance.  But your team needs to be built with specializing generalists in order to optimize for team performance.

T-Shaped People

From an HR perspective, I was taught about “T-Shaped People” – people who have breadth and depth of skills.

• Specialists are “I-Shaped People” – people who have depth of expertise, without breadth
• Generalists are “Minus-Shaped People” – people who have a breadth of skills, but no depth of expertise.
• Specializing Generalists are “T-Shaped People” – people who have depth of expertise in one area, combined with a breadth of skills across many areas.

These are the people you’re going for.

Thanks Elena for re-invigorating the discussion!

After building an understanding of which problems are important to your each customer you want to serve, and rating each competitive product , you’re ready to tally the scores and see how your product compares with your competition.  This tells you if you’re likely to crush it, and if not, lets you know where you should invest later.  This series on comparing products starts here if you need to get caught up.

And now, on to the finale…

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers. (This Article)

With this information, you can create a point of view about how your product compares to the others.

What You Know

During the series, you’ve pulled together a lot of market information, and probably had several insights.  [Reminder so that no one accidentally misinterprets this article - this is a hypothetical analysis with fake data, intended to teach how you do this - it is not an analysis of the Kindle Fire, specifically]  Along the way you’ve determined

Who your target customers are, in order to make sure you design your product for the right people, instead of trying to design for everyone (which is the same as designing for no one).

1. Tina - A hi-tech prosumer who is using the device to get smarter about the latest trends in her industry
2. Tim - A hi-tech prosumer who is using the device to enjoy niche fiction content, particularly comics, e-zines and self-published works
3. Kenny - A typical kindle user who is using the device for his work in the finance space, studying proposals and business plans, etc
4. Karla - A typical kindle user and voracious reader who is using the device to eliminate the large pile of books on her nightstand
5. Chris - A basic consumer who would is studying business in college
6. Christina - A basic consumer who is in a book club, and who is always reading the latest best seller

Comparing Products Part 3 – Market Problems

How important each of those customers is to your strategy, to give you a means of determining the relative importance of each solution.

[larger image]

[larger Blue Ocean image]
[...]
Important Customers – Comparing Products Part 5

Which problems those customers care about solving, to minimize the amount of effort you put into capabilities that don’t effect the purchase decision for those customers.

The importance of each of those problems to each of those customers, so that you know which problem to solve first (or best) for each customer.

1. Read Anywhere – Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.
2. Annotate – Be able to annotate / highlight what I’m reading for future review.
3. Talk About It – Be able to have conversations with other people who are reading what I’m reading.
4. Find More to Read – Make it easier for me to find other content that I would like to read.
5. Subscribe – Be able to subscribe to magazines / newspapers / blogs / serial publications.
6. More From My Network - Be able to read what people I trust are reading.

[larger image]
Comparing Products Part 3 – Market Problems

Your goal is to find the right level of abstraction at which to determine who your competition is.  If we consider Tim, the hi-tech prosumer who consumes niche content, we would see that the decomposition of his high level goals looks like the following:

[larger image]

Know Your Competition – Comparing Products Part 6

5. How effectively each competitive product solves each of those problems, so that you know who your customers consider to be your competition.

Scoring All of the Capabilities for All of the Products

Applying the same process (determine the nature of each capability, determine the criteria for each capability, assign a score to each product for each capability) will result in something that looks like the following [manufactured to illustrate the concepts] data:

[larger image]
Rating Your Competition – Comparing Products Part 7

Now you’re ready to combine this information to see how your product stacks up.

Rolling Up Scores – The Mathy Bits

First, don’t worry – there is a little bit of math here, but it is baked into the process so that you only have to worry about it once or twice, and then it just helps instead of getting in the way.  The math serves to answer several subordinate questions along the way to answering the questions that lead up to “which is the better product, and by how much?”

Cognitive Bias in Customers, but not Robots

For each customer, you know how important each problem is to solve.  You know how effectively each product solves each of those problems.  If your customers were robots, this would be easy – each customer would rationally compute the utility they get from each capability from each product, and optimize their overall utility by picking the product that maximizes this value.  You could just multiply the scores and get the answer.  Here’s one reason why that won’t work (there are others).

We saw with Kano Analysis of realistic more-is-better capabilities, people get a non-linear sense of satisfaction from solving problems, which we reflected in our [made up] data by mapping of the scores (from 1 through 9) against the overall curve for each individual capability.

[larger image]
Rating Your Competition – Comparing Products – Part 7

This approach normalizes the scores within the pseudo-math for comparison. Don’t lose sight of the fact that it is probably harder to improve your product from a “6″ to a “7″ for any one score than it is to improve it from a “2″ to a “3.” What this approach does embody, is that improving a single capability from a “7″ to a “9″ is more valuable than moving from a “5″ to a “7.”

Because your customers are not robots, they will have cognitive biases that cause them to overweight (mentally) their personal “most important problem” relative to the “less important problems” when determining their overall score for a product that solves all of the problems.  In other words, a customer will overweight the “most important capabilities” and underweight the “still important” capabilities, from a purely rational utility-maximization perspective.

It is more important to your customer to be a little bit better at solving their most important problem than it is to be a little bit better at solving their least important problem.

This is true both for robots and customers, and the “just multiply” approach captures this relative importance.  However, when you include the cognitive biases of people, you have to go one step further.

It is more important to your customer to improve your solution to their most important problem than it is to improve your solution by twice as much, to a problem that is half as important.

What the pseudo-math does is change the process from prioritizing by the user’s (robot’s) utility, to comparing products based on the buyer’s perceived utility at the time of the purchasing decision – incorporating the cognitive biases into the way the scores are calculated.

As with any numbers-heavy technique, you run the very real risk of creating your own sense of false precision. This entire exercise is the formulation of a model (in Lean terms, a testable hypothesis) of how to compare competing products.  If you don’t trust (or if you disprove) the approach that I use as a starting point, then change it to reflect what you think (or what you find).

To incorporate this bias that overemphasizes the solution of the most important problems, I square the values of relative importance of problems (to users).  The original data rates the importance of solving problems on a scale from 1 to 5.

[larger image]

When squared, the values become:

[larger image]

I chose to use squaring of the values because it intuitively feels right, when using a 1-5 scale for the (actual) weightings.  There may be a better formula, and the ideal adjustment may be different for each product domain, or even for each persona within that domain.  My gut tells me that if you invest the time in determining the “right” way to account for biases, you will have lost the opportunity to compete in your market, although you will have gained the opportunity to publish a useful psychology research paper.  Use squaring, ignore cognitive bias, or use some other calculation that passes your “sniff test” – it’s up to you.  This is another example of why the Art of Product Management is an art.

Scoring Each Product for Each Customer

When you multiply the scores for each product (for each capability) with the relative importance (of each capability) for each persona, you end up with a table of values that reflect the perceived score for each product for each persona.

For Tina, that table looks like the following

[larger image]

For Kenny, that table looks like the following:

[larger image]

Notice that the ranges of values (for Total score) are different for Kenny and for Tina.  That’s because Kenny cares about fewer product capabilities.  The “Max Possible” column has been added to each table, to reflect the maximum score that each persona would place for any given capability, for any product that scored a “9″ on that capability.  The total-per-product scores for each persona are then reflected as a percentage of the maximum possible score for each persona.  This allows you to manage the comparison across customers effectively.

The table showing the final score per-product, per-customer looks like the following:

[larger image]

What you can see with this view is that none of the products is ideal for any of the personas.  All of the products are pretty good for Kenny, and none of them is very good for Christina, although the iPad 2 is noticeably better than any of the others for Christina.

You can also look at this the other way (columns-first, rows-second), and see that the Nook Tablet is better suited to Kenny than anyone else.  If Barnes & Noble’s strategy involved a focus on Kenny, that would make sense for them. Choosing Kenny may have been a bad idea, however, since the Nook [in our made-up data] doesn’t do a better job – for Kenny – than the other products.

This is the view that is most useful in making product decisions moving forward – making this product comparison process useful to you.  You can use it as a starting point for doing what-if analysis. You can see what you need to improve to improve your product (relative to the competition) for any particular persona.  You can hypothesize about what your competitors might do to improve their products, and what competitive responses you might need.  You can determine if you have a good opportunity to differentiate your product – for a particular persona – or if that part of the market is too crowded.

In practice, you can use this view to (a) assess the viability of a particular strategy (that involves “winning” with a particular persona), and (b) perform sensitivity analysis for introducing new (or improved) capabilities.  To do the sensitivity analysis, start by asking “how much will this improve the rating (for each persona) for the capability?”  Then see what the impact is (in this table) of changing your product’s score(s).  Intuitively, this is asking the question – “Would improving (or adding) capability X turn the dial in the market?”

You can also use this framework to imagine the future positioning impacts of anticipated or likely improvements in competitive products.  This gives you a mechanism for organizing a strategy of competing with what your competitors will be doing (by the time your product launches), instead of trying to compete with where your competitors used to be.

While this is the most useful view for you – informing your individual decisions – you probably need to roll it up to a crisp answer for your boss – answering questions like:

• Which product is best?
• How far ahead (or behind) are we?
• How much do we need to invest to “win”?
• Should we pivot, or should we persist?

Scoring Each Product Across Customers

Earlier in the process, you established a measure of relative importance of each customer, for executing a given strategy with your product.

Using the numbers for the Blue Ocean Strategy, we found [manufactured] the following relative importance of each customer:

[larger image]

When you multiply each per-product, per-customer score by the relative importance of each customer (in a blue ocean strategy), you get a table that looks like the following:

[larger image]

For the Blue Ocean Strategy, the iPad2 and the Kindle Touch are “better” products than all the others, whereas the PC is by far the worst.

If you were to use the Red Ocean Strategy, where we found [created] the following relative importance of each customer:

[larger image]

When you multiple each per-product, per-customer score by the relative importance of each customer (in a red ocean strategy), you get a table that looks like the following:

[larger image]

In the Red Ocean Strategy, the iPad2 is the clear winner, with only the Kindle Touch threatening them.

You can tell your boss, as the product manager for the Kindle Fire:

• We have to pivot away from the Red Ocean Strategy, to the Blue Ocean strategy, if we want to stand a chance.
• We’re still in 3rd place with the Blue Ocean Strategy, but the key to winning it is to win for Tina, where’ we’re pretty much tied.
• The way for us to win with Tina is to exploit the iPad2′s blind spot – finding more to read.
• Here’s my proposal for how we tackle the Finding More to Read user goal for Tina.  In 3 months, we’ll have a better set of features, and 6 months after that the ecosystem around those features will result in us having the better product.

Summary

This is a process for bringing some order, structure, and testability to the process of comparing products – as a tool for informing better future product decisions.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers. (This Article)

With this information, you can create a point of view about how your product compares to other products.

This model has evolved quite a bit since I first used an early version of it in 2009.  Now that you’ve looked at it too – how would you improve it?  When you use it, let me know what you find to be the strengths and weaknesses of this approach.

And thanks for sticking with me over the last 9 weeks, 8 articles, and 16,751 words.  Please let me know what you think, and share this with your colleagues if you think they would value it.

At this point in the product comparison series, you know who your customers are, which problems are important to them, and which products compete to solve those problems.  It’s time to score the competing products and see how the solutions your product provides (or will provide) will stack up.  This is the latest in a series on comparing products, jump back to the start of the series if you came here first, but hurry up :).

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem. (This article)
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Summarizing Effectiveness

Earlier in the series, we identified (and refined) the list of important, relevant problems that our target customers have.

[larger image]

This is the “ruler” by which each competitive product is going to be measured.  We also identified several competitors.

[larger image]

The next step is to assess how effectively each competitive product (including your own) solves each important problem.  Then you need to assign a “score” for how effectively each product solves each problem.  To do that, for each problem you have to articulate an opinion about what it means to solve the problem poorly or completely, or anywhere in-between.

Read Anywhere – Previously clarified as “Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.”

Environments can be indoors/outdoors, extremely cold to moderate to extremely hot temperatures, with variable lighting from a dark room to direct sunlight.  It might also capture environmental context – sitting, walking, riding on a bus, driving, etc.; quiet to noisy; physically serene, or getting bumped a lot (like in a crowded coffee shop).

Start by defining the endpoints.  I’ve been using a 9-point scale in this type of analysis, to provide enough granularity to make relative comparisons.  For read anywhere, a score of 1 would mean “can be used to read in a single, idealized environment / location.”  A score of 9 would mean “can be used to read in any realistic situation.”  Mapping out the scores in-between 1 and 9 requires you to think about the nature of the problem being solved – and here’s where Kano analysis is useful (again).

This capability is a good example of an extreme more-is-better capability.  Increasing the range of environments where the product can be used (to read) provides a perceivable benefit to customers, but with diminishing returns.  Also, there is some minimum bar, or table stakes, of environments where the user needs to be able to read, or the product is not considered a viable solution.  On the high end, being able to read literally anywhere, would truly distinguish one product – making that capability a delighter and a strong differentiator.

[larger image]

How do you decide “What is a 3 score?” You inform these relative scores based on user research (ideally), and your subjective opinion (when you don’t have research).  For folks who haven’t been reading Tyner Blain articles for the last few years – a product manager is market driven, which means you need to use market data to do this as a product manager; however, using your own opinion as a product designer is better than having no data at all.

For this example, my [manufactured, invented, made up for this series of articles,] data defines scores for the Read Anywhere capability as follows:

1. Below The Bar - User must be seated, and have power and internet connectivity (at the time of reading) in order to use the device.
2. na
3. Usable but Very Annoying – User can read while standing without being connected to power or the internet.
4. Not Quite Happy, but Whatever - User must be indoors, with reasonable lighting and temperature.
5. Meh - User can read while riding on the bus or in a car.
6. OK, but Nothing Special- User can read in outdoor lighting.
7. Good - User can read pretty much anywhere except really noisy, jarring, and / or low-light environments.
8. na
9. Wow! - User can read anywhere that the user would want to read.

Note that there are no entries for 2 or 8, to reflect that there’s a step-function decrease (or increase) in perceived value at this point in the curve.  Plotted on the Kano analysis extreme more is better curve, this rating scale looks like the following:

[larger image]

One of the things you can see when looking at the scoring system visually is that you may have to make significant investments in order to make incremental improvements, depending on where you are on the curve.  For example, moving from (4) to (6) looks hard – you are objectively increasing the capability measurably – and increasing the subjectively perceived value by a comparable amount.

A small shift in scoring – for example, from (7) to (9) – can have a dramatic impact on perceived value (specifically, satisfaction received from improving the capability).  Conversely, improving from (6) to (7) may be really hard (expensive) to do, but realistically only improves the way your market perceives your product by a marginal amount.

This view can help you answer questions like

• Why does the iPod shuffle outperform the Sansa Clip so dramatically? Because the iTunes-centric ecosystem turns the dial from (7) to (9).
• Why doesn’t our detergent outsell theirs, when ours gets clothes cleaner in soft water?  Because moving from (6) to (7) doesn’t make much of a difference.

Zooming in on the low-end of the curve:

[larger image]

And at the high end of this capability, we see:

[larger image]

Applying this rating scale to the competitive products [more made-up data here] we see

[larger image]

It may be that different personas would use markedly different criteria for scoring relative capability.  So far, when I’ve done this, I’ve found that different persona care different amounts about the scores for a particular capability, but generally use the same approach to scoring.  When I do come across different personas that would use markedly different scoring criteria for the same capability, I will create use the appropriate scoring criteria for each persona, and add more complexity to this process.  To date, I haven’t had to do that.

Scoring All of the Capabilities for All of the Products

Applying the same process (determine the nature of each capability, determine the criteria for each capability, assign a score to each product for each capability) will result in something that looks like the following [manufactured to illustrate the concepts] data:

[larger image]

Alternate Scoring Method

You could also approach determining the score for a single capability like this by giving points for each characteristic, versus trying to define a continuum like the example above.  For example, you could give

• 2 points for being able to use the device without power.
• 1 point for being able to read when you are not connected to the internet.
• 1 point for being able to read in bright sunlight
• 1 point for being able to read in a dark room
• etcetera

Then tally up the score for each product, to describe how well they meet the need that users have to read anywhere.

Interpreting the Comparison

If you were to stop here, you would just conclude that the iPad2 is the best, and that the two Kindle products are “close,” and that the Nook and using a PC are in last place by a wide margin.  However, you aren’t going to stop here.

Stopping here would ignore completely that different customers care different amounts about solving different problems.

In the next article in this series, we’ll see how to incorporate that info, and answer the two questions you need to be able to answer:

1. Which product is best for a specific persona?
2. Which product is best overall.

Summary

To create a competitive product, you need to know how your product stacks up against the competition – and that means you need to know how effective your solutions are (or are perceived to be) at solving the problems your customers will pay to solve.  You can compare today’s product to assess your current position, and you can inform the decisions about what your product needs to be.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem. (This article)
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Taking it to the next level, as a product manager, your decisions about tomorrow’s product should be in the context of where you expect tomorrow’s competition (and tomorrow’s customers) to be.  There is a danger – especially after investing in the “state of the industry” analysis above – that you will continue to compete in yesterday’s race, when you should probably be innovating to redefine the game.  The comparison you just did is not a waste (because it looks at today’s problems – they become the table stakes for tomorrow).

Remember, this exercise informs future product decisions, it does not define them.

Attributions

Thanks Rick Cox for the height comparison photo.

You start with a point of view about what makes a minimum viable product.  When your product launches, it is your customer’s point of view that matters.  You must understand which problems your customers care about solving, and what solutions are available to your customers today.  You need to understand your competition to make informed decisions about your product.  This is the latest in a series on comparing products – jump back to the beginning of the series to catch up, we’ll wait.

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition. (This Article)
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

No Competition

If you think your product has no competition, you’re wrong.

The photo above shows an El Camino, a combination of a car and a truck.  The front of the vehicle, the cabin interior, and the overall height / profile of the vehicle is that of a car; but the vehicle has a tailgate and a truck bed for hauling stuff.  A pretty novel idea at the time.  There were no other car-truck combination vehicles, but that doesn’t mean the El Camino had no competition.

You should not define competitive products as the products that exactly match the features of your product.  You should not define competitive products as those that solve (exactly) the same set of problems that your product solves.  Every interesting problem already has a solution – and therefore, you have competition.  It may be that there are not any particularly good solutions in the market.  It may be that the competitive products are not well known, and would not be likely to compete effectively.

It may be that the solution your customers choose is “deal with it.”  The “deal with it” solution, at first glance, from a B2C (business – to – consumer) perspective feels like “there is no competition,” but I believe that embracing that point of view puts you at risk of being myopic about your market.  In the B2B (business – to – business) area, “deal with it” is one version of the “in house solution” competitor – a very real scenario.  Stacking ridiculous amounts of stuff on top of your car is analogous to managing your accounts receivable in a spreadsheet.  You can do it, but it is a solution that is fraught with other risks.

Your customer generally has four options from which to choose when deciding if they should buy your product.

1. Buy your product.  Hooray.
2. Buy someone else’s product.
3. Build their own solution.  Your customer believes that rolling their own is a better alternative to purchasing a solution from someone else.
4. Deal with the problem.  The cost of the solution is believed to be higher than the cost of the problem.

How do you identify which products are compelling competitors?

Problem Abstraction

One thing you learn as a product manager, when eliciting requirements, is to keep asking “why?” (and more and more on asking “why?” – check out the comments on these articles).  The goal in elicitation is to find the actual underlying problem, and not just the obvious manifestation of the problem.  One technique you can use is to create an Ishikawa Diagram that helps you define the problem(s) your product is attempting to solve.

Your goal is to find the right level of abstraction at which to determine who your competition is.  If we consider Tim, the hi-tech prosumer who consumes niche content, we would see that the decomposition of his high level goals looks like the following:

[larger image]

There are several benefits to using this approach of describing the decomposition of Tim’s goals:

1. You get traceability of your requirements, allowing you to see why each goal matters, from Tim’s perspective.
2. The graphical layout forces you to be concise in how you describe Tim’s goals.
3. You have a framework for validating that you have defined the complete set of requirements needed to satisfy Tim.
4. Looking at “everything” in one view helps you assure that your requirements are consistent.
5. You know “when to stop” in decomposition when your requirements are atomic.
6. You have a framework for assuring that you focus on those goals that Tim is passionate about.
7. You have tractable goals that you can use to think outside of the box when identifying competitors.
8. You have a clear and visceral tool for communicating with your team, to inspire innovative designs and clever solutions.

Finding Competitors

Some of the competitive solutions will be very straightforward, and obvious based on your understanding of your product domain.  Others you will discover through research.

When we look at the Kindle Fire as a product, we know from Amazon’s positioning that Tim will be able to read books & magazines (in color), consume multimedia content, browse the internet, stream movies, and borrow books.  He’ll have access to “millions of books,” be able to “read anywhere,” and connect with people over email from the device.  The Barnes & Noble Nook Tablet is positioned directly as a competitor to the Kindle Fire (and can even be purchased through Amazon).

It is occasionally annoying to read reviews of the Kindle Fire that basically say “not an iPad killer,” where the reviewer compares the features of the Kindle Fire, a single purpose device with multiple capabilities, with the iPad, a multi-purpose device that can be used for (many) single purposes.  The iPad is a viable competitor for the Kindle Fire, but a comparison in the other direction does not make sense.  The products are asymmetric substitutes – they are only substitutes for each other in a particular context.  If you are Tim, trying to enjoy niche content, then the iPad is a viable competitive product.  If you are a 15 year old boy who wants to play a first person shooter when you’re being shuttled to soccer practice, the Kindle Fire is a non-starter.

Finding a needle in a haystack is easier than identifying competitive products.  You know there is only one needle – once you find it, you stop looking.

Identifying competitive products is more akin to finding out how many needles are in the haystack

You can do research on the internet – start with the positioning from already identified competitors, look for editorials, rumor articles, reviews, research papers, customer reviews, discussion forums.  Search social media streams – find conversations (and broadcasts) about the product and the domain.  This gives you either minimal or overwhelming information, depending on how many competitors are already positioning themselves as products in this market segment.

Interview people who share goals with your personas – they don’t have to match your personas (but they can) – you are trying to discover solutions that you don’t yet know about.  Find out how they address those goals.  Maybe they use a web browser on their desktop computer, cobbling together a cabal of single-purpose solutions.  If they already own a competitive product, find out what they did before they bought that product.  As an example, someone like our Tim persona, before buying a purpose-built product, did the following:

• Used The Sword and Laser podcast to discover new content and get recommendations.
• Used Facebook to get and share suggestions with his friends, and have conversations about his favorite works.
• Used wikipedia and artist individual websites to find other works by his favorite artists.
• Subscribed by email to artists that published serials, blogs, other recurring content.
• Purchased tangible copies of works from Amazon.com.
• Borrowed tangible copies of works from IRL (in-real-life) friends.
• Purchased ebooks in pdf, epub or other formats that could be read on his computer and phone.
• Used Dropbox (or drag-and-drop by USB) to synchronize content across devices.

The problems that people are solving did not come into existence when the solutions were invented. The problems predate the solutions. [Note:if you have a Forrester subscription, check out Mary Gerush's Exploit the Requirements Lifecycle for a more detailed discussion].

The roll your own solution is a viable “competitor” for a B2C product.  In the B2B world, this is usually called best of breed. [Note: it is usually called this by one of the vendors of one of the pieces of the cobbled-together solution].  Using these amalgam-of-products solution strategies can make sense – and for some customers it will be the best solution.

We know that Christina and Time (two of our personas, defined earlier in the series) are different personas – they value different goals differently.  They will also use different approaches to create their own mix and match solutions to their problems.  Previously, I said that you didn’t have to interview people who represented your personas in order to identify competitors.  You do, however, need to figure out which product bundles represent the roll your own solution for each persona.  That is best done by interviewing representative customer prospects.

Our Example Competitors List

In this series, we are showing the mechanics of performing a product comparison, not sharing an actual analysis of the Kindle Fire.  As such, this is (again) where I get to manufacture illustrative data.  Here’s the list of competitive products that could have resulted from the research and analysis described above:

1. The Kindle Fire
2. The Nook Tablet
3. The Kindle Touch
4. The iPad 2
5. A laptop running windows, with a pdf reader, the free kindle app, and an HTML 5-compliant web browser.

The list above is not exhaustive.  Like most analysis activities, you can go on forever, but with diminishing returns.  Where you draw the line is a matter of risk mitigation.  Add as many competitors as you feel you need to accurately assess how well your product (or a future version of your product) will compete in the market.  There is a risk associated with both too much analysis and too little analysis.

Too much analysis slows you down and is expensive.  You will get a comprehensive view, and minimize the risk of missing something, but you run the risk of taking too long.  You also run the risk of never stopping – you can never find all of the needles in the haystack – and therefore you run the risk of never shipping or shipping too late.

Too little analysis saves money and time, but you run the risk of delivering the wrong product.  If you happen to identify all of the competitors that have (or will have) success in the market, you’ll be sufficiently informed (because your view will be a super-set of your customers’ views).  If you overlook a product that is driving the market, you run the risk of watching the world go by through rose colored glasses – you’ll watch people buy that other product, and not understand why they aren’t buying yours.

Here’s where agile provides a perspective that lets you escape the catch-22.  If you do “too much” you can never recover.  The money you spent is a sunk cost.  The time you lost is gone.

If you do too little, you can always do more later, when you realize you need to, as long as your team is capable of embracing change.

Summary

To create a competitive product, you need to know how your product stacks up against the competition – and that means you need to know who the competition is.  A product marketing manager may take this information to emphasize strengths and “turn weaknesses into strengths.”  A product manager can use the insights from this analysis to paint an informed picture of what the product needs to be (and why).

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition. (This article)
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Attributions

Thanks Matthiew Yglesias for the El Camino photo.

A good product is one that solves valuable market problems.  To be successful in the market, a product needs to solve the problems that the right customers are willing to pay to solve.  To know if those customers are willing to pay, you need to understand how they perceive your product relative to alternative solutions.  If you’re new to the series, head back to the intro article on comparing products, and catch up with this article, where we look at pulling together the information about which customers are important.

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers. (This article)
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Target Personas

Previously in the series we identified (and refined) a set of personas that make sense for comparing products that compete with the Amazon Kindle.  [Note: for people new to the series, this is a hypothetical analysis - using manufactured data - with the goal of showing the mechanics of how to do the analysis, not an actual analysis of real market data.]

1. Tina - A hi-tech prosumer who is using the device to get smarter about the latest trends in her industry
2. Tim - A hi-tech prosumer who is using the device to enjoy niche fiction content, particularly comics, e-zines and self-published works
3. Kenny - A typical kindle user who is using the device for his work in the finance space, studying proposals and business plans, etc
4. Karla - A typical kindle user and voracious reader who is using the device to eliminate the large pile of books on her nightstand
5. Chris - A basic consumer who would is studying business in college
6. Christina - A basic consumer who is in a book club, and who is always reading the latest best seller

[...]
We also identified a set of problems that these personas would want to solve.

1. Read Anywhere – Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.
2. Annotate – Be able to annotate / highlight what I’m reading for future review.
3. Talk About It – Be able to have conversations with other people who are reading what I’m reading.
4. Find More to Read – Make it easier for me to find other content that I would like to read.
5. Subscribe – Be able to subscribe to magazines / newspapers / blogs / serial publications.
6. More From My Network - Be able to read what people I trust are reading.

[larger image]
Comparing Products Part 3 – Market Problems

You can’t create great solutions for all of these problems, for all of these customers, at the same time.  This is one of the two main failings of the bureaucratic approach I’ve seen large companies take (the other one is using a long waterfall process, that even if it succeeds, results in a product that solves yesterday’s problems by the time it is released).  The old process would have you gather inputs from multiple stakeholders, identify multiple target customers and the capabilities needed to support them, and roll it all up into a giant document with MoSCoW reflections of the importance of each capability.  Since every capability is important to someone, every capability ends up being “important.”

If you’ve worked on a non-agile project, you’ve seen this:  Everything is high priority.

Except it isn’t.  The problem that the monolithic document, waterfall processes does not take into account is that while everything may be important, not everything is urgent.

Only items that are both important and urgent are high priority.

Every problem identified in our example is a “five” to at least one of our personas.  If you’re trying to be all things to all people, they are all important.  Figuring out which ones are urgent requires you to form a strategy on which problems you need to solve first.

Solve for One Persona First

This process is informed (biased?) by insights that resonated for me from the writings of Seth Godin, Al Ries & Jack Trout, Geoffrey Moore, Hayden Christensen, and innumerable conversations with other product professionals.  In summary, it has bubbled up into the following perspective from me.

• A product that is great for some people, even if unusable for other people, is a great product.
• A product that is usable by many people, but great for none of them, is a bad product.
• If you wait until your product is “perfect,” or even “great for a lot of people,” at best you will deliver yesterday’s product and no one will care anymore.

Believing this, my approach then is to ask:

• For whom should we build first?
• What do those customers expect and what will they love?

Those expectations are set, in part, by your competition – your market is a moving target - which is why comparing products is important to figuring out what problems to solve first, and how well you need to solve them.

Another way to think about it – success in your market is a reflection of aggregate sales.  But sales don’t come in aggregate.  Sales are made one at a time.  In any given sales conversation, your customer prospect is comparing your product with your competition.  If your customer has two choices – the first being marginally interesting (even if it happens to be marginally interesting to “everyone”), the second being “perfect,” which one will the prospect choose?  For any given sale, you want the reasons why that customer would purchase your product to be overwhelming.  You want to be the product that is perfect, not the product that is tolerable.

Even if your goal is to sell to “everyone,” you can’t launch your product with it being the “perfect” product for everyone.  You can launch with a focus on a single persona.  Note that I didn’t say with a product that is “perfect” for a single persona.  You probably don’t need to wait until it is perfect to start selling it.  You certainly shouldn’t wait until it is perfect before introducing it (and getting feedback that helps you improve) to people that are represented by your target persona.

When you think about the minimum viable product (MVP), the minimum is not “be tolerable to everyone” – it is “be good enough for someone.”  Focus first on one group of people.  And figure out where your minimum bar is.  That minimum bar is defined by a combination of

• Which problems are important to solve – if you don’t solve the “showstopper” or “table stakes” problems, you don’t have an MVP, and won’t even be part of your customer’s decision process
• How well do you need to solve each of these problems – the next article in this series gets into this, but it is primarily a matter of matching or exceeding customer expectations, which are informed (or anchored by) how well competing solutions solve those problems today.

Picking the Right First Persona

There are a bunch of factors that can feed into the appeal of targeting any particular persona first:

• Market Size - Fish Where the Fish Are.  All things being equal (although they never are), the largest market is the most attractive market.  How you define “size” may influence which market you pick however.  Is it the market with the potential for the most units sold?  The one market with the highest revenue potential or the one with the highest profit potential?
• Cross-Market Influence - After you’ve launched, you need a plan for growth.  Do you grow by gaining market share with your target persona, by growing share of wallet (sell more stuff) with your existing customers, by growing into adjacent markets (other personas).  Does picking one target persona first, and making them rabid fans, help you transition into other markets?  The best way to sell me a gadget is to convince the digerati (The Verge, Engadget, etc) that it is the best product ever.  The best way to sell my mom a gadget is to convince me.
• Higher-Level Strategy – Your product may only be part of a portfolio of products and services.  Your company may be focused on owning a niche, like being the technology supplier to all health care professionals.  The strategic value of being a “full service provider” or of preventing the entry of a competitor into a market segment by satisfying the personas in that segment may dwarf the value of any siloed value of your product.
• Risk Mitigation – This is the Lean approach.  With which group of customers can you most quickly, most cheaply, and most effectively test your hypothesis that you have a great product that will satisfy needs and succeed in a large market?  Early adopters are the most forgiving technology customers, and often provide the most insight about what problems the larger markets will care about later.  There’s also implicit value in the “gee whiz” factor of novel solutions to existing (and new) problems.  You can put out a beta version of a product to early adopters, where they are ok with some bugs, workarounds, and missing capabilities.  This isn’t likely to be your end goal – you’re serving these people first in order to more effectively serve another market next.
• Ease of Entry – Nature Abhors a Vacuum.   There may be a persona for whom none of the existing (competitive) products provides particularly compelling solutions.  You may want to attack this segment first, because easy early sales can fund your company and keep the lights on while you tackle the larger, and likely more competitive, market.  I think that one of the reasons the Innovator’s Dilemma happens is because there is a niche market with unsolved problems, into which companies introduced products that eventually (through improvements in the product, and changes in the needs of the larger market) becomes competitive and eventually displaces the original leaders in the larger market.

Jason Cohen, founder of Smart Bear Software, wrote a great piece on finding the “perfect customer” – an example of the risk mitigation approach.  The conversation on Jason’s article is great too – and I love one of his comments: “There’s no such things as ’5x better’ if there’s no customer in mind. Because “better” is in the eye of the beholder.”

Your overall approach will determine the different weightings you apply to each of these factors.  As a couple straw men for this example, Amazon might be taking a red ocean (beat the competition in an existing, defined market) or a blue ocean (“create” a new market by solving different problems) approach.  Those approaches might yield the following initial relative weightings:

For the personas in this example, I’ve made up the following reflection of their relative attractiveness along each axis.

[larger image]

These 1 to 5 values reflect the relative attractiveness of each persona for each strategic approach.  For example, the size of the potential market of people like Christina is much larger than the size for any of the other personas.  Note that the Ease of Entry field is shown in orange versus green to emphasize that these initial values reflect a guess at the ease of entry – a general perception of how effectively existing products address the needs of these personas for the problems we previously identified.  When you do the more detailed analysis in future steps, you should circle back and revisit these values.  In Marketing Warfare by Ries and Trout, they point out the importance of the linkage between strategy and tactics – each informs the other.  A tactical understanding of your market (like an understanding of the competitive products)) will inform the best strategy – the strategy that comes from an ivory tower will fail in the field.  That philosophy is consistent with the iterative process of refining a comparison of products, as part of informing your strategy.

The strategic approach you choose will impact the “pseudo-math” of which persona is most important.  Consider the two examples – when you use the percentages from the first two tables to apply a weighting to the assessment of each strategy aspect, you get the following:

[larger Red Ocean image]

A Red Ocean Approach would indicate that Christina is the most important persona on which to focus.  The combination of the large market size of basic consumers reading for entertainment, along with an overall strategy to focus on the broad market makes Christina your target.

[larger Blue Ocean image]

A Blue Ocean Approach would indicate that Tina is the most important persona on which to focus.  The perception that no other products address her problems dominates the other factors.  Once we confirm that no one is solving the market problems that we know are important to Tina, we will have market data that validates that a blue ocean strategy is viable.

There are a lot of numbers here, and definitely a risk that you let the numbers overwhelm decisions and drive everything mechanically, like a computer algorithm.  That would be bad.  Where this data-driven view is useful is in giving you a framework for questioning the strategic approach.  The “math” highlights that a strategy of going toe-to-toe with competitors, and attacking the largest market first says Christina is the persona on which you should focus.  A blue ocean approach would drive you to emphasize investment in solving the problems that are important to Tina.  You also get a feel for how many “happy accidents” you might get along the way, by also “accidentally” creating a great product for Kenny & Chris while focusing on Tina, because the problems that Tim cares about also happen to be ones that Tina cares about.

[larger image]

Remember, though, that the Ease of Entry initial guesses will change as soon as you pull together detailed information about how effectively competitive products address the problems that are important to each persona.  Next we’ll look at the competition.

Summary

To create a competitive product, you need to know which customers on whom you should focus and for which problems they would value solutions.  There are always more potential customers, with differing needs and perspectives, than you can build product to satisfy.  You need to figure out which customers to focus on, and which ones you consider a “happy accident” if you acquire them.  Even if your management team insists that your product is “for everyone,” you need to know for which customers you will solve best and deliver first.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers. (This article)
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Attributions

Thanks Jon Worth for the skater photo.

Thanks Larry Lamsa for the skateboarder photo

If you understand the important market problems, you can make a good product.  If you understand how important each problem is, for each group of customers, you can make a great product.  If you’re new to this series, go back and start at the first article, we’ll wait for you right here.

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers. (This article)
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Important Problems

In the previous article on defining market problems, we identified 6 personas / contexts by which we would compare the kindle fire with other products.

1. Tina - A hi-tech prosumer who is using the device to get smarter about the latest trends in her industry
2. Tim - A hi-tech prosumer who is using the device to enjoy niche fiction content, particularly comics, e-zines and self-published works
3. Kenny - A typical kindle user who is using the device for his work in the finance space, studying proposals and business plans, etc
4. Karla - A typical kindle user and voracious reader who is using the device to eliminate the large pile of books on her nightstand
5. Chris - A basic consumer who would is studying business in college
6. Christina - A basic consumer who is in a book club, and who is always reading the latest best seller

The reason it was important to identify contexts is that the important aspect of personas is to identify groups of users with homogeneous perspectives on the relative value of solutions to particular problems – and the three personas previously identified would place different values on the solutions, depending on the context in which they were using the device.

Revisiting the Market Problems

We also identified a set of problems that these personas would want to solve.

1. Read Anywhere – Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.
2. Annotate – Be able to annotate / highlight what I’m reading for future review.
3. Talk About It – Be able to have conversations with other people who are reading what I’m reading.
4. Find More to Read – Make it easier for me to find other content that I would like to read.
5. Subscribe – Be able to subscribe to magazines / newspapers / blogs / serial publications.
6. More From My Network - Be able to read what people I trust are reading.

This process is iterative, and in reviewing the 6 problems above, a valid question is – are problems (4) and (6) different versions of the same problem?  When writing requirements, it is important to specify the problems and not the design.  This is a tricky one, as it blurs the line between requirements and design.  Reasonable people can make either of the following interpretations:

• The market requirement is to “find more to read” by any means necessary – it could be through receiving recommendations from the user’s network, or it could be based on some not-specified “black box” heuristic and scoring algorithm.  These two requirements are duplicates.
• Yes, ultimately the user is trying to find more to read, but this is actually too abstract – consider the goal of “enjoy using the device,” which is also the ultimate goal, and too abstract to be useful in guiding the product creation.  Asking people you trust for recommendations is a well established practice for finding reading material, and people make the distinction that it is inherently different from, and provides unique value relative to other approaches (like finding similar products, or “people who bought this also bought this other thing“).

Based on research I’ve done for previous clients (primarily on findability and recommendations in an eCommerce context), my personal perspective is that these problems are distinct, and should be treated differently.  This is one of those Art of Product Management situations, where different product managers can, will, and should reach different conclusions.  For this analysis, the problems will be treated distinctly, and the data that I invent will reflect that I believe these problems would manifest with different importance for different personas in different contexts.

In contrast to the above analysis, problem (1) conflates the two notions of “multiple devices” and “multiple environments.”  People familiar with eInk technologies and Amazon’s Whispersync technology may see these as independent solutions to reading in direct sunlight, and moving from device to device respectively.

I’ve combined these two ideas, primarily because of the influence of this presentation about multi-screen strategies from Precious , a design and strategy consultancy.

[image from Precious article]

My interpretation is that the notion of device shifting is something that is intentional – use the right device for the right environment – and not arbitrary.  Based on that, and the technologies that enable reading in different physical environments – like direct sunlight or a dimly lit room – should be evaluated as part of implementing a device-shifting strategy, and not independently.

Combining these two (in this example) is an opinion-based decision, just as keeping problems (4) & (6) distinct is an opinion based decision.  If the data you gather in identifying problems leads you to combine or separate those problems, do so.

Gathering the Data

In the previous article on identifying market problems, I indicated that you are using a mix of qualitative and quantitative data-collection techniques to identify the problems.  It is during those activities that you also quantify the relative importance of solving these problems.

Innovation Games are a particularly engaging way to gather this type of information [disclosure: while I haven't formally done work for the company, I have worked with and know and respect the founder Luke Hohmann , and likely will work with him again in the future].  I’m including them here because they have worked for me.  A couple examples:

• 20/20 Vision – get customers to put solutions to the problems into relative order (for them).  Effectively, a card-sorting exercise.
• Speedboat – the relative priority insight you get from this game comes when people put anchors at different depths, indicating the severity of a particular problem.
• Whole Product – this brainstorming game can also be used to identify what people perceive to be Must Be (table-stakes) capabilities of your product.

Surveys can also be used to gather data when you need to get feedback from larger numbers of people, or are operating in an environment that is less collaborative.  A simple approach – ask the question “How important is it to you to solve problem X?”  Read up on Likert scales to understand the dangers of asking this type of question, and determine if you need to bring in someone who is an expert at survey design to minimize the impact of bad survey design in your results.

Quantifying the Results

Using the word, quantifying, may not be wholly accurate here – it implies something scientific, when really you are synthesizing the data you have received, in order to eventually determine the relative importance of solving (or improving your solution of) specific market problems.  A made-up word like numberifying might be better.  If quantifying, used in this context, causes your head to explode, please comment below – I suspect this might be a hot-button for folks, but I’m not really sure if it is.

[larger image]

The above table shows hypothetical data representing the “quantified” relative importance of having a solution for each problem, to each persona / context.

For example, Kenny wants to use a device to review and annotate proposals and business plans – internal company documents.  He can do this on any of a number of devices, but the ability to do it “anywhere,” as long as he can annotate, is what really matters to him.  He’s not at all interested in capabilities to find more to read, discuss what he’s reviewing, or subscribe to new content.

For Christina, however, the act of reading is more social than private.  She is not worried about making notes in what she reads or subscribing to new content – she’s reading books and talking about them with her friends.

Gaining Insights

We’re not to the point in the series where we can actually compare competing products effectively yet, but you can already start to get insights.  Imagine you are designing a product for Christina.  Your business case is about going after the “Oprah book club” audience, and Christina is your target user.  You already know which problems are important to her, and more importantly, which problems are not important to her.  You can completely eliminate the subscription and annotation capabilities from your product and create something that Christina would love.

This step alone allows you to avoid the elastic user problem.

Usually, you aren’t able to design a product for just one user.  You can’t be all things to all people.  This is part of the analysis you have to do to determine what to build first – identifying the relative importance of problems to each persona.  The next step is to figure out the relative importance of each persona.  Combining the two allows you to understand the overall relative importance of each problem.

Summary

Creating a great product for your customers means not only knowing who your customers are, but also knowing which problems they want to solve, and among those problems, which ones it is most important to solve or solve first.  When comparing your product to competitive products, the best measure is one that compares the products based on the most important problems for each group of customers.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers. (This article)
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Comparing products without an understanding of the important market problems by which to compare the products is a waste of time.  This is the third in a series on comparing products - jump back to the introduction if you haven’t already read the previous articles.  Go ahead, we’ll wait, then come back.

Overall Product Comparison Process

This is a relatively long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems they care about solving. (This article)
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Too many product comparisons are actually mislabeled positioning papers, highlighting differences, and putting one product in the best possible light.  An honest product comparison must compare products in the context of problems that customers are willing to pay to solve.  Those are the important market problems.

Important Market Problems

Probably everyone reading this has heard the Henry Ford quip about how if he had listened to his customers, he would have built faster horses.  A great marketing sound-bite, but not actually true.  The market problem that cars initially addressed wasn’t “get there faster”, the market problem was “spent fuel” disposal (for horses) in major cities – 2.5 million pounds per day in New York City (hat tip to Super Freakonomics for the first reference I saw).  If cars had generated as much manure as horses, do you think they would have been as successful?

The first step in identifying the important market problems is to identify a set of market problems that might be important. Then you can form an opinion about which ones are important, based on which problems are important to the potential customers in the markets on which you are focusing.  Identifying problems is an unconstrained activity, like collecting seashells on a beach.  There will always be more shells.

You will find shells that aren’t very pretty or are broken.  You will find some shells that are “perfect”, and there will always be some shells that exist, that are “perfect” that you won’t find.  The tough part is knowing when to stop looking for shells and move forward with the ones that you’ve already found.  You have to decide when your shell collection is “good enough.”  You can’t have a collection that includes all of the seashells, you don’t want a meager collection with too few shells, and you want the best shells you can find to be the ones you display on your coffee table.

This is a good example of the debate between big up-front design (BUFD) & big up-front requirements (BUFR) and incremental product development.  In The Design of Design, Frederick Brooks provides a great discussion about this debate between rationalism and empiricism.  The best approach I’ve found to date is to use a combination of agile and waterfall techniques to strike a balance between good enough and right now.

The best approach that I’ve found to do this to combine incremental development techniques to the activities of product management, combined with timeboxing to impose practical, artificial constraints on the process.  An agile product management process looks roughly like the following:

[larger version]

The steps of market research, analysis, and synthesis are the activities you perform.  Data is what you collect, and insights are what you form.  Analysis of data also leads you to perform more research.  Synthesis of those insights into a cohesive vision for your product will lead you back to more analysis and more research.  Note that this process has no ending – it goes on forever – or for as long as you are trying to solve these particular market problems.

As an example – in the previous article we identified a set of personas for our Kindle Fire example:

For this series of articles, I will pretend that three personas were developed, in order to demonstrate this approach to comparing products:

1. Hi-Tech Prosumers – people who get excited about convergence, highly value additional multimedia capabilities, live a multi-device existence, and are acutely aware of competitive products.
2. Typical Amazon Kindle Users – people who placed high value on the reading experience, the convenience of being able to easily get new content, and the absence of a subscription fee.
3. Basic Consumers – people who read primarily on paper media, have recently started consuming multimedia on their computers, and anticipate becoming part of the “post pc era.”

I will also invent additional data as we explore the rest of the product comparison process.
Comparing Products Part 2 – Who Are Your Customers?

However, defining requirements for those personas risks creating a product that only satisfies the basest of goals – the lowest common denominators.  These examples don’t take into account the context of use – e.g. why is someone reading on their device? – and thus we are at risk of not creating the right product for any of them.  Even introducing two simple contexts

• Education - Reading for work or school (directed study) or to gain knowledge about a topic (undirected study)
• Entertainment - Reading for pleasure, the literary equivalent of watching a movie

You’re probably not going to want to take notes on the uses of foreshadowing and allegory when reading for entertainment.  It may be critically important to annotate key concepts and ideas when reading for education.

This single distinction should be clanging a bell in your head – it is very easy for one product to be better than another in one context of usage than in the other.  Looking at the three example personas from the previous article, there is not a good way to say “educational use is more important” for one group of people than for another.  If we iterate and refine our list of personas it will lead to a better comparison of products.

1. Tina - A hi-tech prosumer who is using the device to get smarter about the latest trends in her industry
2. Tim - A hi-tech prosumer who is using the device to enjoy niche fiction content, particularly comics, e-zines and self-published works
3. Kenny - A typical kindle user who is using the device for his work in the finance space, studying proposals and business plans, etc
4. Karla - A typical kindle user and voracious reader who is using the device to eliminate the large pile of books on her nightstand
5. Chris - A basic consumer who would is studying business in college
6. Christina - A basic consumer who is in a book club, and who is always reading the latest best seller

Identifying Possibly Important Problems

Now that we have a (better) idea of the customers for whom we are solving market problems, we need to create a list of candidate problems that might be worth solving.  I am not an expert on user research – so there may very well be a better way to do this, and hopefully someone will share that with us in the comments section – but here’s how I’ve learned to do it.  It is at least “good enough.”  The approach I’ve used is a 3-step process.

1. Seed the Cloud – Start with what you think you know already
2. Refine the List – Validate and enhance that list through qualitative interviews
3. Empirical Validation – Assess the relative importance of those problems through quantitative research

Seeding the Cloud

I like to start by capturing my initial ideas in a mindmap or concept map – an ad hoc, semi-structured view of ideas.

[larger image]

I will then drive some brainstorming (more on brainstorming) or idea-seeding or Innovation Games exercises to develop a refined list of market problems that might be worth solving.  These processes may lead to other artifacts and usually result in updates to my mindmap.  These exercises bring others into the exploration, so that it isn’t The World According to Scott – but it is still the world as seen “from the inside.”

Refine The List

What we’ve done at this point, while creating a set of questions worth asking, possibly, is still heavily biased as an inside-out view of the market.  It may be informed by previous research and synthesis of ideas, but it still reflects what we think about the market, not what customers in the market thinks about their problems.

The next step is to gather qualitative information from customers and potential customers that represent your target personas.

Anecdotally, this is the same process for driving outside-in innovation from a product management perspective.

Mostly-open-ended conversations with people that are reflective of your personas is how I transform from an inside-out view to an outside-in view.  I’m not asking people “what they want” (remember the faster horse), I’m getting validation that the problems they face are the ones I think I understand (waste disposal).  This usually uncovers additional problems I have not thought of, and discounts problems I may have thought were important.  Ethnographic research provides this type of information – and when I’m working with a team that does this work, I utilize it.  Usually, I’m not, so I will find representative users and ask them questions, or observe their behavior.  Essentially, I’m playing the 80/20 game – some insight is better than no insight.

Empirical Validation

At this stage, you still only have a list of possibly important market problems, but now you have much more confidence that the list is valid and mostly complete.  You also have some qualitative insights into the relative importance of problems to be solved (maybe you did a card-sorting exercise, or created interrelation digraphs, or played the 20-20 Vision game, to get those inputs).

Now you can gather empirical data, to gain some statistical significance to your analysis.  You can do surveys that help you characterize and profile user behavior.  You can even do analysis of social networks, to see which problems resonate and why particular products get good (or bad) word of mouth.

Amplified Analytics has a presentation on slideshare showing how they can mine statistically significant information about features in the context of products. [Hat tip to Gregory, for the pointer to this].

I would suggest using a similar approach to try and understand which problems people face (and solve) versus which features they prefer (as solutions).  But the same applies.  I’ve had good results creating surveys and using Google Docs to aggregate results that allow me to characterize problems, (reported) behavioral patterns – like frequency of use, and (reported) relative importance of solving particular problems (like performance times for specific actions), and quantifying data-characteristics (e.g. how many emails someone sends, and to how many different people, in a given day).

Example Problem List

Given that we’re not actually doing a rigorous (and free) analysis of the Kindle, I get to “pretend” that I did research following the examples above, resulting in the following list of market problems, that are of (at least some) interest to our target personas.  We’ll use this list moving forward in the rest of the series.

1. Be able to read content in multiple physical environments / on multiple devices, and not lose my place in the book.
2. Be able to annotate / highlight what I’m reading for future review.
3. Be able to have conversations with other people who are reading what I’m reading.
4. Make it easier for me to find other content that I would like to read.
5. Be able to subscribe to magazines / newspapers / blogs / serial publications.
6. Be able to read what people I trust are reading.

Summary

It is important to be creating a product with an identity.  That means knowing for whom the product is designed, and for which uses.  You have to know who your users are, and which problems they want to solve.  If you’re going to compare products, you should compare them based on their suitability as solutions to these important problems.

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers.
3. Articulate the problems your customers care about solving. (This article)
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Attributions

• Thanks Stéphane Vandenwyngaert for the Model T photo.
• Thanks michaelaw for the smiling coffee drinker photo.
• Thanks Dora Pete for the self-portrait photo.
• Thanks Benjamin Earwicker for the spontaneous photo.

The first step to comparing products is understanding your customers.  This may seem counter-intuitive, but your product’s capabilities are meaningless unless you are comparing them from your customer’s point of view.  This article is part 2 in a series on comparing products.  Check out part 1, then continue with this article on the first steps of comparing products.

Overall Product Comparison Process

This is may be a long series.  Each article will start with a recap of the overall process.

Getting useful information from comparing products requires you to:

1. Introduction & Overview (so that the step-numbers align with the article numbers)
2. Identify your customers. (This article)
3. Articulate the problems they care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

From that point of view, you can determine where to invest in your product.  That’s the whole point of doing the comparison – not to be self-congratulatory, but to guide forward thinking.  I’ve stopped being excited when I find existing “product comparisons” and game plans when starting to work with a product team.  To date, they have always been “positioning papers” designed to emphasize the product’s strength – usually prepared for internal sales teams, but unfortunately, sometimes prepared for executives (‘Look!  We’ve done a great job!”) Those documents are a useful source of information to get you started down the product comparison path, but if you let them guide you, they will take you down the primrose path.

Identify Your Customers

You want to have a model that represents how you are approaching your market. As a product manager focused on solving customer problems, you should have a customer-centric market model.

Your goal is to identify groups of customers that share sets of problems, and share a point of view about how important it is to solve those problems.  Effectively, this is a definition of a persona (yes, I know there are better ways to define personas).

The important thing is that you identify groups of people that will reach the same conclusion about your product, by answering the following questions similarly -

• Which problems that are important to me can I solve with this product?
• How well does this product solve those problems?
• How does this product compare to other products?

Personas, Goals, and Contexts

The image above is adapted from Robert W. Bailey’s diagram from Human Performance Engineering, (although I first saw it in the Handbook of Usability Testing).  One use of Bailey’s diagram is to understand that to assess the usability of a product, you have to take into account the human being who is using the product, the environment in which they are using the product, and the actions the person is trying to perform.

My extensions of the idea are that

• Groups of people with similar sensibilities (personas) will form similar conclusions about how much they like a product when,
• They are using the product in similar contexts,
• To achieve similar goals

One benefit of abstracting this way is that you don’t get caught up in the mechanics or procedures of how someone is solving a problem (the actions).  Focus on the action is appropriate when evaluating the usability of a particular product, but not when comparing how different products solve the same problem.  I also like something that reminds me that people use products in different environments and situations.

Example Personas

Usually, when I see personas they are focused purely on marketing and demographic data, or they are based solely on contextual inquiry intended to inform a design aesthetic.  The former may tell you where to market a product to groups of people (magazines, tv ads, luxury travel websites, etc), and the latter informs how to implement solutions.  What we need is something that helps us assess the relative importance of solutions to problems shared by groups of people.

This series of articles is looking at the Kindle Fire as a proxy product by which I will explain the techniques (without actually performing a data-driven analysis – I’ll invent data for the purpose of this article series).

I found a presentation on Slideshare:

BlackBerry Consumer Preferences and Segmentation

Where some MBA students documented the research and analysis of the market of prosumers – “professional” consumers.

What I particularly like is the way they presented a summary of part of their analysis on slide 27.  That’s the part we’re going to pretend applies to the Kindle Fire (because it might just).  Note that their presentation includes a “RIM Confidential” clause in the footer, so it might have been removed by the time you read this article.  I’ll briefly describe what they did.  They did an analysis of survey results, identified groups of people that placed similar importance on different activities and aspects of smartphone usage, and identified three “clusters” of people with statistically significant shared sensibilities.

1. Hi-Tech Prosumers – people who placed high value on most features, anticipated high usage of most features, were price-sensitive, had lower incomes (less than $75K USD per year) and were the younger members of the target demographic.
2. Typical Blackberry Prosumers - people who placed high value on network features, moderate value on other features, made heavy use of “standard” capabilities, but did not plan to use apps, or multimedia.
3. Basic Prosumer - people who placed high value on phone features, were heavy users of text and voice features;  had low use of other capabilities, but anticipated becoming heavy users of “smartphone capabilities” (email, calendar, browsing, apps, camera, etc) when they purchased their next phone.

What’s notable is that they did the right type of analysis – finding clusters of people that can serve to quantify market value of different personas – this will inform the determination of the relative importance of different groups of customers (step 3 in the process of comparing products).  By interviewing individual survey respondents that are in each of the clusters, they can get the additional information they need to develop personas.

For this series of articles, I will pretend that three personas were developed, in order to demonstrate this approach to comparing products:

1. Hi-Tech Prosumers – people who get excited about convergence, highly value additional multimedia capabilities, live a multi-device existence, and are acutely aware of competitive products.
2. Typical Amazon Kindle Users – people who placed high value on the reading experience, the convenience of being able to easily get new content, and the absence of a subscription fee.
3. Basic Consumers - people who read primarily on paper media, have recently started consuming multimedia on their computers, and anticipate becoming part of the “post pc era.”

I will also invent additional data as we explore the rest of the product comparison process.

Developing Personas

Please don’t finish this article believing that persona creation is trivial – it isn’t.  I’ve heard the saying before that a bad persona is better than no persona at all, and I believe that.  Without a target persona, you will almost certainly create the elastic user anti-pattern.

Persona creation is more of a precondition of this process than a part of this process – you should already have personas developed for your product.  This article is not intended to go into the details of how to create a persona, but rather a review of the aspects of that persona development that are needed for the product comparison process to be effective:

• Identification of people in your market that share perspectives on problems – e.g. knowing that for one group of potential customers solving problem X is important and problem Y is not, even when problem Y’s solution is important to another group of customers.
• Being able to estimate the relative number of people that are represented by each persona – e.g. knowing how much revenue potential exists for sales to each group of people.

Summary

Recapping the overall flow of this series of articles on product comparison

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers)
2. Identify your customers. (This article)
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to other products.

Attributions

* Thanks Alaska Dude for the Lovely Lass photo.

Recently, the gadget-reviewer crowd has caught on to something we’ve known for a long time.  Comparing products is not about comparing specs, it is about comparing how well the products solve problems that customers will pay to solve.  That begs the question – how should you compare products?  Read on to see the product comparison technique I recommend.

Inspiration

Writing about how to compare products has been on my backlog for about two years, as a key component of how to perform competitive analysis.  This topic is easily a chapter-length discussion, perhaps that’s what delayed writing an article-length discussion.  A flurry of recent articles, The Death of the Spec, Do Device Specs Really Matter Anymore, and Device Specs, from Techcrunch, iDownloadBlog, and Daring Fireball, respectively, all discussed how the Amazon Kindle Fire will likely succeed, in spite of not having particularly good specs (specifications).  The Techcrunch article also opines that Consumer Reports, because of its fixation on specs, has made itself irrelevant as a company providing advice to consumers.

John Gruber hits closest to the mark in the Daring Fireball article where he says

Specs are something the device makers worry about insofar as how they affect the experience of using the device. Just like how focal length and lens aperture are something the cinematographer worries about insofar as how they affect what the viewer will see on screen.
John Gruber

Combine this with a conversation I had last night after recording the latest Start With the Customer Podcast, about writing more frequent articles on Tyner Blain.

Product Management and Specs

The discussions are mostly centered on the utility of specifications, speeds and feeds, in informing a customer’s buying decision.  As Pragmatic Marketing espouses, we should be market-driven, with a focus on solving the problems that customers will pay to solve.  A screen resolution does not solve a problem, but an easy-to-read screen does – it prevents eye-strain and makes a long-session reading experience (like you would have when reading a book, versus reading this article) better.  Avoiding eye-strain is a problem people are willing to pay to solve – we’ve seen that with the success of products that use e-Ink technology.

Let’s look at how this example impacts what you do as a product manager.

You want your team to create a product that avoids eye-strain.  But you also know that you need to write requirements that are unambiguous and measurable.  We know from research that higher resolution displays (to a point) delay eye fatigue.  We also know, from Apple’s successful marketing, that promotion of a retina display (326 dpi, for a phone) is effective, at least with buyer personas, at addressing the perceived problem as well.

A “normal” consumer is not going to be able to make an informed comparison of the likely difference in eye strain over time – the problem the consumer actually cares about – when looking at the specs for a 225 dpi device and a 250 dpi device.  The authors of the articles are exactly right about that – but we, as product managers, already know this.

The problem comes when writing the requirements for your product team – do you just say “create a low eye-strain screen” and trust the team to pick a resolution that is effective? In an ideal world, yes, you would.  Someone (that someone may have to be you) on your team would do the research and come back and tell you that a 225 dpi interface will cause moderate eye-strain in 80% of people after 12 hours of continuous reading (but only 20% of people after 4 hours), and that that number drops to 20% of people experiencing eye strain after 12 hours when using a 250 dpi screen**.  You have an understanding of the value of a 250 dpi resolution over a 225 dpi resolution – an additional 8 hours of reading time for the majority of your customers.  Your customers will not know this (unless you tell them).  But you know it, and that’s enough.

Now you have to understand the incremental cost of creating a product with a 250 dpi resolution versus one with a 225 dpi resolution.  In this example, the incremental device cost is $25 per device for the next 12 months (given projected manufacturing levels).  At your target margins, this would reflect in an increase of $40 in device pricing to your customers.

This higher-capability, higher-priced product will simultaneously appeal to more customers (people who read for more than 4 hours at a time), and fewer customers (people who are price sensitive).  Your hypothesis (backed with market-research) indicates that you will generate 50% more profit by offering the lower-capability resolution (225 dpi), because most of the people in your target market do not regularly read for 12 hours at a time – and those that do will not “blame” your product for their eyestrain – they will “blame” their own behavior.

Now you’re ready to to specify that your product will be built with a 225 dpi resolution interface.  Not because a 225 dpi interface is in any way intrinsically valuable, but because it is the measurable and unambiguous requirement needed to satisfy your goal (achieving profitability) based on a solution to a more is better (see more articles on Kano analysis) market problem (eye strain that occurs in long reading sessions).

Specifications are useful, because they help you characterize capabilities.  The diagram above depicts a more is better characteristic, as described in Kano analysis.  The resolution measurements (dpi) give you a measurable criterion for what you are building, that translates into a measurable criterion (hours of continuous use without eye strain) that your customers will realize.  That criterion reflects the horizontal axis of the diagram.  The longer your customer can read without straining her eyes, the more capable your product is.

The vertical axis reflects how much your customer cares about solving the eye-strain problem.  Note that there are diminishing returns.  Enabling 4 hours of use (without eye strain) versus 2 hours of use is more highly valued than enabling 6 hours of use versus 4 hours (or 8 versus 6).

The above analysis compared the selection of discrete components, but it also applies when looking at incremental investment.  The speed at which the screen refreshes when turning pages is a good example.  You can have an e-Ink screen that refreshes in 1 second, or one that refreshes in 0.5 seconds, each time your user turns the page.  There is no difference in incremental cost, and your team is operating with a fixed budget of time and resources – so there’s no impact on the allocated fixed costs (or margins).  You are faced with a different set of compromises – what are you willing to give up (by reducing investment in other aspects of your product) to achieve incremental improvement in page-turning responsiveness?  It may be that page-turning snappiness is the one idea of your product (it was a differentiator for the Barnes & Noble Nook, but now the e-Ink Kindle has achieved parity).  Or you may be dealing with one of the “host of others” capabilities, in which case you simply want to satisfice.

Bigger Picture

The above example shows how specs matter (indirectly) in the everyday lives of product managers.  Specs are the measurements by which we evaluate the effectiveness of our products at solving the problems our customers care about.

If you are designing a product that has no competition – for customers that have no alternatives, this would be enough.  It would be more than enough, actually, because you could create “any” product, and it would sell.  Your customers always have alternatives – so you always have competition. Sometimes, your competition is “build your own” or even “tolerate the problem.”  But for any interesting market problem, you have at least one competitor trying to solve it too (or you will.  Very soon).

Comparing Products Matters

As a product manager, you need to know what your product needs to be (or do) to be competitive.  That’s where comparing products matters.

The above analysis looks at a single problem (eye strain) for a single product (yours), to try and determine what specs to give to your team.  In a competitive environment (that means you), you need to put the effectiveness of your product in context from your customer’s point of view.  That involves identifying

• Who are your customers and what problems do they care about solving?
• How important, relative to each other, are the solutions to those problems, to your customers?
• How important, relative to each other, is each group of customers?
• What solutions (products) do your customers consider as alternatives (competition) to your product?
• How effective is each product at solving each problem?

From that information, you can synthesize a point of view on how competitive your product is (or will be).

That point of view helps you identify

• Which problems you need to invest in solving, or solving more, or solving more effectively.

Summary & Series

Specification is not a good tool for helping non-expert customers compare products.  It is, however, a good tool – an unambiguous measurement tool – that product managers can use when specifying how a product should be built, or how one product compares to another.

Products must be compared based on their effectiveness (and perceived effectiveness) at solving problems that customers will pay to solve.  Those comparisons should also take into account the relative importance of those problems, as well as the relative importance of different groups of customers, to the success of the product.

Even at 1600 words, this article barely introduces the topic of comparing products.

Recapping the overall flow of this series of articles on product comparison (I’ll update this article with links to future articles in a series on comparing products.):

Getting useful information from comparing products requires you to:

1. Introduction and Overview (so that the step-numbers align with the article numbers) (This article)
2. Identify your customers.
3. Articulate the problems your customers care about solving.
4. Determine how important solving each problem is, relative to the other problems, for your customers.
5. Characterize how important it is for you to solve the problems of each group of customers.
6. Discover which (competitive) products your customers consider to be your competition.
7. Assess how effectively each competitive product solves each important problem.
8. Assess how effectively each competitive product solves each important problem, for each important group of customers.

With this information, you can create a point of view about how your product compares to the others.

Attributions & Notes

* Thanks Roger Wollstadt for the original Volkswagon Assembly photo

** The dpi-related eyestrain statistics are fictional, and written to demonstrate the importance of measurement only

Your boss wants a commitment.  You want to offer a prediction.  Agile, you say, only allows you to estimate and predict – not to commit.  ”Horse-hockey!” your boss exclaims, “I want one throat to choke, and it will be yours if you don’t make a commitment and meet it.”  There’s a way to keep yourself off the corporate gallows – estimate, predict, and commit – using agile principles.

This is an article about agile product management and release planning.

Change and Uncertainty

In the dark ages before your team became agile, you would make estimates and commitments.  You never exactly met your commitments, and no one really noticed.  That was how the game was played.  You made a commitment, everyone knew it would be wrong, but they expected it anyway.  Maybe your boss handicapped your commitment, removing scope, lowering expectations, padding the schedule.  Heck, that’s been the recipe for success since they planned the pyramids.

It makes sense.

1. Your early estimates are wrong. When you add them up, the total will be wrong.  If you do PERT estimation, the law of large numbers will help you in aggregate.  But you’ll still be wrong.
2. The outside demands on, and availability of, your people will change. Unplanned sick time, attrition, levels of commitment over time, lots of “people stuff” is really unknown.
3. The needs of your customers will change. Markets evolve over time.  You get smarter, your competitors get better, your customer’s expectations change.

Agile processes are designed to help you deliver what your customer actually needs, not what was originally asked for. Contrast the two worlds.

In the old world, you would commit to delivering a couple pyramids. After spending double your budget, with double the project duration, you would have delivered one pyramid. When you deliver it, you find out that sphinxes are all the rage. Oops.

Your team changed to agile, so that you could deliver the sphinx. But your Pharaoh still wants a commitment to deliver a couple pyramids (the smart ones will be expecting to get just one).  You can stay true to agile, and still mollify your boss’ need to have a commitment, if you take advantage of the first-principles of why agile estimation works.

Estimation

A commitment is a factual prediction of the future.  ”This will take two weeks.”  Nobody is prescient.

A factual prediction has to be nuanced.  ”I expect* this will take no more than two weeks.”

*in reality, this is shorthand for a mathematical prediction, such as “I expect, with 95% confidence, that this will take no more than two weeks.”

Few non-scientist, non-engineers, non-mathematicians understand that 95% confidence has a precise meaning.  People usually interpret it to mean “a 5% chance that it will take more than two weeks.”  What it really means is that if this exact same task were performed twenty thousand times (in a hypothetical world, of course), then nineteen thousand of those times, it would be completed in under two weeks – do you feel lucky?

To make a statement like this, you actually have to create a PERT estimate – identifying the best-case, worst-case, and most-likely case for how long a task will take.

Unfortunately, we’re rarely asked to make a commitment about a single task – but rather a large collection of tasks – well-defined, ill-defined, and undefined.

You can combine PERT estimates for the individual tasks, resulting in an overall estimate of the collection of tasks.

The beauty of this approach is that the central limit theorem, and the law of large numbers, work to help you estimate a collection of tasks – you can actually provide better estimates of a group of tasks than a single task.  This obviously helps with the well-defined tasks that you know about at the start of the project.  This even helps with the ill-defined tasks.  Rationalists will argue that the key, then, is to do more up-front research to discover the undefined tasks – and then we’re set.  As Frederick Brooks (Mythical Man-Month) points out in The Design of Design, this debate has been going on since Descartes and Locke.  It is not a new idea.

Big Up-Front Design and Requirements (BUFD & BUFR) hasn’t worked particularly well, so far.

Don’t throw out the baby with the bath-water, however.  The math of estimation is still important and useful.  Even if empiricism is not the silver bullet.

Prediction

Estimation is a form of prediction.  Even agile teams do it.  In Scrum, you estimate a collection of user stories – in story points that represent complexity, and you predict how many points the team can complete in this sprint.  Note the time factor.  If you’re working a two-week sprint, there is very little risk of changes in staffing during a two-week period.  There’s also very little risk that your market will change significantly in two weeks – and if it does, what are the odds that you will notice and materially change your requirements in two weeks?

Visually, let’s take that PERT estimate and turn it sideways – so we can introduce the dimension of time.  Imagine you estimated all of the tasks (well-defined, ill-defined, and a guess about the undefined), as if they were all to happen in the first sprint.  Ignore inter-task dependencies, and pretend you had unlimited resources and the ability to perform all tasks in parallel.

The graph above shows the aggregate estimate – the circle is your best prediction, with error bars representing your confidence interval in the estimate.  If you were using PERT estimates, these could represent that 5% and 95% confidence lines.  Subjectively pick something based on your team’s experience in the domain and your confidence in your guesses (about the undefined tasks).

We need a segue into the “best of waterfall” approach to estimating projects, to steal and invert a good idea.

The Cone of Uncertainty

The folks at Construx have published a nice explanation of the cone of uncertainty – an adaptation of an idea from Steven McConnell’s Software Estimation: Demystifying The Black Art (2006).  That article uses his imagery with permission – so please go look at it there.  The idea is that as the project becomes better defined (e.g. during the project), the amount of uncertainty is reduced.

The findings show that initial estimates are off by 400% (either low by a factor of 4 or high by a factor of 4)!  Even after “nailing down” requirements, estimates are still off by 30% to 50%!

As bad as that sounds, it is actually worse.  This is a prediction for the original project (delivering pyramids).  Not only are your estimates wrong – but they are bad estimates for delivering the wrong product.

But – the core idea is sound – the further into the future you have to execute, the greater the mistakes in your estimate.

Taking that concept, and applying it to our diagram, we get the following:

The further into the future you are trying to predict, the less accuracy you have in your prediction.  This reduction in accuracy is reflected as a widening of the confidence bands for your estimate.

• A couple sprints’ worth of work is not much different than one sprint – so your estimation range is not much changed.
• An entire release of sprints (say 6 to 10 sprints) has much more opportunity for the unknown to rear its head.

Now, your prediction is (probably) unusably vague and imprecise.  ”This set of tasks will take X plus or minus a factor of two.”

That’s the reality.

Note: This has always been the reality.  People have historically reduced this “risk to timing” by hiding the “risk of change” aspects – and waterfall processes encourage you to deliver the wrong thing, as close to on-time as possible.

That’s not what we want to do, however.

We still want to deliver the (not-yet-defined) right product, as efficiently as possible.  That’s the goal of agile.  (For folks who haven’t been here at Tyner Blain for long – “right” includes both value and quality).

Refinement

Because we’re agile, and we’re willing to “get smarter” about our product over time, we have an opportunity to improve.  Because of the nature of compounding estimates and the cone of uncertainty, our uncertainty gets smaller over time.

Let’s remove our artificial simplification that we could do everything “right now” and look at what we think we know right now, about the end of the release.

Our ability to predict the amount of effort (for today’s definition of the product) at the end of the release is not very good.

Our ability to predict (today’s definition of the product) one sprint into the future is much better.

After completing the first sprint, we are a little bit smarter – the ill-defined tasks are better defined.  Maybe some of the undefined tasks are now ill-defined.  The same cone of uncertainty is now a little bit smaller – we are a little bit smarter, and the time horizon of the release date is a little bit closer.

The trend continues – each sprint gets us closer to the release date, and with each sprint (assuming we get feedback from our customers, and continue to study our markets) we get a little bit smarter.  We also get better at predicting the team’s velocity (how much “product” they can deliver during each sprint).

Commitment

Your boss still wants a commitment, however.  And that’s where we get to change the way we look at this (again).

The above diagrams all display how we converge on an estimate for a stable body of work.  However, we know that the body of work is constantly changing.

Backlog! [you say]

Yes!  The backlog.  The backlog is an ordered, prioritized list of user stories and bugs.  I was talking with Luke Hohmann of Innovation Games last month, and one of the most popular online Innovation Games is now the one they created based on prioritizing by bang for the buck.  Play it today online (for free!).  How cool is that?

The backlog represents the work the team is going to do – in the order in which the team is going to do it.  Over time, as we get smarter, we will add and remove items from the backlog – because we discover new capabilities that are important, and because we learn that some things aren’t worth doing.  We will even re-order the backlog as we recognize shifting priorities in the markets (or in our changing strategy).

As this happens, it turns out that the items at the top of the list are least likely to get displaced, and therefore most likely to still be part of the product by the time we get to the release.

Instead of thinking about uncertainty in terms of how long it takes, think about uncertainty in terms of how much we complete in a fixed amount of time. In agile, generally, we apply a timebox approach to determining what gets built.

Now, uncertainty, instead of manifesting as “when do we finish?” becomes “what will we finish?”

Your boss is rational.  She appreciates the constraints, she just wants to know what you can commit.  Every boss I’ve worked with has been willing (sometimes only after much discussion) to treat this uncertainty in terms of what instead of when.  They acknowledge that they need to translate (usually for their boss) into a “fixed” commitment.

The solution: commit to a subset of what you predict you can complete.

At the start of the release, you may have 500 points worth of stories.  Based on your team’s expected velocity, and the number of sprints in the release, you predict that you can complete 320 points worth of stories (5 people on the team, a team velocity of 40 points per sprint, and 8 sprints in the release).  Starting at the top of the backlog and working down, draw a cut-line at the last story you can complete (when you reach 320 points).  This is your prediction.

Now the commitment part.  You’ll have to figure out what you’re comfortable with.  Maybe for 8 sprints (say, 16 weeks into the future), you may only be comfortable committing to half that amount – 160 points.  Go back to the top of the backlog, and count down until you reach 160 points.  Everything above the line is what you commit to delivering.

Maybe you are comfortable committing to 240 points, maybe only 80.  This is like playing spades.  The more you can commit to, without missing, the better off you are.  Your tolerance for risk is different than mine.

You can also negotiate with your boss.  Commit to 160 points now, and provide an update after every other sprint.  More likely than not, you will be increasing the scope of your commitment with every update.

Mid-project updates of “we can do more” are always better than “we can do less.”  And both are better than end-of-project surprises.  This also allows you to have updates that look like this:

We didn’t know this at the start of the release, but X is really important to our customers – and we will be able to deliver X in addition to what we already committed.  Without slipping the release date.

Conclusion

Making commitments with an agile process is not impossible.  It just needs to be approached differently (if you want to stay true to agile).  The end result: better predictions, more realistic commitments, and the likelihood that each update will be good news instead of bad.

[Update: Changed initial image.  Thanks Dennis for the great photo!]

Requirements Management – I’m embarking on a journey to help several teams manage their requirements with their existing systems and tools.  This is the first in a series of articles, where the rubber meets the road.  I’ll look at both the theory and the realities of what works (and doesn’t) in practice.  I hope you’ll come along for the ride.

The Idea That Makes it Tricky To Manage Requirements

OK, so I’m working with a product management team that is working with multiple software development teams.  The development teams already have tools and systems in place for tracking work within projects.  The development teams are mostly using Atlassian’s JIRA for tracking activity.  The teams also have Atlassian’s Confluence wiki for capturing “other stuff.”

The development team has a good working process for managing their delivery activities within the “JIRA world.”  From the top down view, you find a project (in JIRA) that represents what will be delivered in a particular release.  Within that project, you see the issues (or user stories or requirements) and associated tasks that were previously scheduled for that release.

This is pretty effective for getting a handle on what’s happening.  Where it falls short is in getting a comprehensive understanding of why it is happening.  And that’s the view product managers need.

Why something needs to happen is completely different from when something needs to happen.  That’s what makes this tricky.  Consider the following analogy.

You’re driving across the country from Los Angeles (LA) to Manhattan (NYC).  Your goal is to get to Manhattan.  Your product manager creates a map, breaking down, roughly, how you will get from LA to NYC.  Your product manager is riding shotgun in the car, and he’s responsible for telling the driver where to go.  Your developer is driving the car, and is responsible for getting the car wherever the product manager tells him.  They share the responsibility for getting to NYC – neither of them can do it alone.

They climb in the car, and start driving.  When the developer has a question about a particular way to go (which highway to take, etc), they communicate.  Together, they make great progress.  After a while, the gas tank is empty, and they stop to fill up.  This is a logical break in the drive, and they stretch their legs, get some dinner, whatever.  With a full tank of gas, they get back in the car and start driving again.

Managing requirements inside the context of a release (in JIRA) is like saying every time you stop for gas (finish a release), you should create a new map.  The goal (destination) is independent of the release cycles (gas tanks).  The requirements should live outside the release.  However, the turn-by-turn information is useful inside the release.  The product manager needs to provide the next level of detail (first, get to Las Vegas, then stay at the Westin,…) for each “tank of gas” – after which, that information is not very valuable, and can be discarded.  But the goal (reach NYC) stays alive.

A process and system for managing requirements should not force product managers to recreate documentation for each release.  Your stakeholder (perhaps you’re delivering a donated kidney to a hospital) needs to know when you’re going to deliver the kidney.  The last place they should have to look for that information is within the context of the current project.

This is what makes it tricky.

JIRA and Confluence

This is not a series of articles around picking the best requirements management solution available today.  These articles will cover the exploration of using JIRA and Confluence to make (1) the product management teams more effective, and (2) the combined product delivery teams (management, marketing, development, quality) more effective.  We’re faced with a real-world constraint that we will use these tools that are already in place.

Product Management Goals

There are a series of goals that we have, that will inspire our implementation choices (of how we use Confluence and JIRA), and ultimately provide the measure of our success.

• We will not remove or regress things that are working today.  The implementation teams (development and test) are delivering products, and using JIRA to track issues and tasks.  Whatever we do must not break this current world.
• As product managers, we are continually investing in, and updating, our understanding of our markets.  We need a solution that allows us to record that information in a way that we can (a) share with each other effectively, (b) remember what we need to remember, (c) make changes as needed so that the view we have documented is always reflected of the understanding we have “in our heads.”
• As a delivery team, we need to connect the two worlds (“what am I doing” and “why are you doing it”), so that the product managers can let the implementation team know what to do next (and what they will eventually be working on).
• As a company, we need to set and manage expectations – internally to stakeholders and externally to customers and partners.

These are the problems we’ll be tackling.

This series of articles will capture elements of implementation choices, why we made them, and the rationale “behind the whys.”

I hope you’ll come along for the ride.