This post introduces the thinking behind our 1-day UX workshop on user research and strategy. (An earlier version appeared at res.im on May 26, 2016.)

The greatest risk in any project isn’t going over budget, delivering late, or changing the scope. The greatest risk in any project is building something people won’t use.

This isn’t just a high-magnitude risk, it’s also highly probable. Failure to deliver value to users happens more often than many people realize. When someone does take a more rigorous look at value being delivered they usually find that “nearly everything fails.”

In one internal review, Microsoft engineers found that only about ⅓ of feature ideas had statistically significant positive results when tested with users. Another ⅓ of the ideas had neutral results, while ⅓ of the assumed-to-be-good ideas actually produced negative results.

Even where Lean and Agile methods are used, it could take months before the product or feature is exposed to users. In the meantime, time and money are spent on assumptions that could easily be refuted or refined with a bit of research or prototype testing.

Say you’re thinking of adding a new feature to your product or service. You might feel confident based on its popularity in other apps—e.g. chat, “stories” or photo filters/lenses—but what works for one audience and context might not work for yours.

Identifying and testing assumptions doesn’t need to turn into analysis paralysis or months of costly studies. When done well, approaching ideas with a bit of research-style discipline saves more time than it costs.

Before doing anything, ask a few key questions to get a sense of how risky your assumptions are. This is how we figure out how much research and validation is worthwhile.

1. Who will use this?
2. When will they use it (or what will cause them to want or need it)?
3. What value or benefit will they get as a result of using this?
4. Is this really the most efficient and effective way to deliver that value?

There’s a fifth question, which might be the most important. It often requires asking a few different ways before getting a good answer.

• How do we know these answers?
• How are we testing our assumptions?
• What results will tell us we’re right or wrong?

Spending a few hours reframing assumptions as testable hypotheses might mitigate months of uncertainty and disagreement. A few days of guerrilla research might achieve the same clarity and alignment as weeks worth of deliberation. Or a one-week design sprint might lead to a strategic pivot that would otherwise require months of development time.

Avoiding tough questions only delays inevitable answers. Skipping research doesn’t actually reduce research costs, it turns the whole project into a research cost. When approached openly, early questions and answers lead to more successful products sooner.

Sign up for our 1-day UX workshop to learn more about how to apply these principles to user research and strategy.

(An earlier version appeared at res.im on May 26, 2016.)

The Basics

Setting the Rules & Making Picks

There are some different ways you could run the selection and scoring process (search for “golf pool” or “Masters pool”) but a golf pool doesn’t really need to be complicated: pick a bunch of players and whoever’s players get best scores wins.

The format we settled on for this pool was to each draft 7 players, from which we’d only count the top 4 scores. There were only two of us, and neither of us is a PGA expert, so we had some fun and mixed in a lot of sentimental, half-serious picks. (We also did this very last-minute—Thursday night, actually—so we had the benefit of picking some people who were already playing well.)

We listed our picks in a very basic Google spreadsheet. I manually entered each of the players’ scores, and used SUM formulas to add them up. Easy enough so far…

Adding More Functionality

Importing Live Scores

As simple as the initial spreadsheet was, I didn’t feel like manually updating scores all weekend. Luckily Google Sheets has a bunch of IMPORT functions that make it fairly easy to pull data in from other sources. Once the live data is in the spreadsheet you can do all kinds of creative stuff.

The IMPORTHTML function is what I used here. It might require a little trial-and-error to find a good source, but once you find one it’s an easy way to pull data from another website. In this case the URL I used was ESPN.com’s Masters leaderboard. You also need to know whether you want to pull data from a table or list on the page. In this case it’s a table. Lastly, you need a numeric reference to the table or list you want to pull from (don’t overthink this one; it’s just a number, which you can guess).

Just type the IMPORTHTML formula with those three pieces of information — “http://www.espn.com/golf/leaderboard”, “table”, 1—into a single cell and a whole table or list should magically appear.

Looking Up Individual Player Scores

The main thing I wanted to do was use the live data to update the scores for each of our picks. The most common way to do this is the VLOOKUP function. VLOOKUP is one of those things that involves a little more thinking but comes in handy once you get the hang of it. (More advanced users running lookups in large tables can combine INDEX and MATCH for more flexibility and faster processing.)

If you’re a spreadsheet novice I encourage you to take full advantage of the tooltips and help available in Google Sheets, and don’t forget to copy or autofill it to the other cells.

The VLOOKUP formula above is basically saying:

• I want to find the value of cell B3 (“Lee WestwoodL. Westwood”) from within a range of cells…
• The range I want to look in includes the “Player” and “To Par” columns of the data we imported from ESPN.com (in this example that is Data!D2:E189)—to which we add dollar signs (Data!$D$2:$E$189) to keep it from changing when we copy or autofill it to the other cells.
• I want to return the value from 2nd column of that range (which in the example above is Lee Westwood’s “To Par” score of -1). FALSE means that we’re looking for an exact match (don’t overthink this).

You probably noticed that I had to change the players’ names to the weird format that appears in ESPN’s live data import. That’s easy to change, along with some other things we need to clean up to make the data more accurate and usable.

Cleaning Up the Data

The biggest issue so far is that the scores aren’t adding up. This is because a lot of ESPN’s data is actually text, so math formulas skip right over it. Specifically, we want to get rid of the “+” sign for over-par scores and change even scores from “E” to the number zero.

It’s often a good idea to clean up data before doing anything else, but in this case I just focused on the data I was using. The SUBSTITUTE function got rid of the “+” signs and changed “E” to a zero. It took a little trial-and-error to figure out how to make multiple substitutions using a nested formula. Even then, the numbers were still coming out as text, so I added a VALUE function to convert the scores to numeric values.

For the player names I simply added new columns and typed them the way I wanted them to appear in our final results, which we’ll get to shortly…

Sorting and Adding the Top Scores

Remember we only wanted to count the top 4 scores from each team. If we were updating the spreadsheet manually we could select the data we want to sort, click Filter in the Data menu and sort each table. But we’d have to keep doing that every time the scores changed. Fortunately, setting it up to sort automatically is just as easy using (you guessed it) the SORT function.

The SORT function creates a copy of the designated range/table (e.g. B3:E9 for Team A) and sorts it by whichever column you indicate (e.g. 4 for the numeric scores in column E). TRUE tells it to sort in ascending order, from lowest to highest (which in golf is best to worst).

Once the scores are in order you can use a couple of good ol’ SUM formulas to add up the top 4 scores for each team. I did this by adding new columns and merging cells (select the cells you want to merge and select Merge from the Format menu).

Making it User-Friendly

What we have so far technically does everything we need but it doesn’t look great. It only takes a few minutes of extra work to create a scoreboard that’s actually somewhat ok to look at—adds a layer of protection over the underlying calculations.

It’s generally good practice to separate your scoreboard or “Results” view from the underlying operations and data tables. It gives you a lot more flexibility to play with the the visual design while keeping the Calculations and Data logical and usable in a utilitarian way.

Notice we’re only showing the normal player names in our Results tab, and the “+” signs and “E”s have reappeared. That’s because the calculations have already been made in another tab, so we’re free to show scores in text format here.

For spreadsheet beginners, if you want cell C6 in the Results tab to display whatever is in cell C13 in the Calculations tab, click on C6 and type “=Calculations!13” — or just type “=” and click on cell C13 in Calculations.

A lot more could be said about formatting and visual design, but this example should be pretty straightforward. Visual spreadsheet design might be a topic for a future post.

Finally, don’t forget to lock the important sheets! Right-click on the tab you want to lock down and choose Protect Sheet. Google Sheets let’s you prevent or warn certain people (including yourself) from making any changes—just like it lets you decide who gets access to view it.

Check out the live spreadsheet here: https://docs.google.com/spreadsheets/d/1Mg_3MYJUdFFWTGDp-loZyU2HszQlKyyaXdQmR_D-o-k/edit?usp=sharing

After roughly 15 years of frustration I finally learned that these doors are what some people call “Norman doors,” named after Don Norman, one of the pioneers of user experience design. The first chapter of his classic Design of Everyday Things talks a lot about doors. There’s also a good video by Vox & 99% Invisible (below). Both are worth your time if you design things or just feel strongly about how things are designed.

Being a UX researcher and usability tester, I wanted to confirm whether other people had the same problem I have with these doors. So I decided to do an informal usability study last September during Country Music Week. The event was a chance to observe “new users” from out of town who hadn’t already learned to use the doors “correctly” (though after speaking to more people it seems a lot of us still struggle).

This was far from a rigorous study, but it only took 15 minutes to convince me that I‘m not the problem, the doors are. It’s a small sample, but the need for more data declines when you watch people struggle with something as basic as pushing a door open. Results were basically 50/50 for push vs. pull—not a great success rate.

Here are the results from that 15 minutes, watching from a nearby bench:

• 5 people “correctly” pushed the door open.
• 5 people “incorrectly” pulled the door handle, then figured out to push.
• 2 people grabbed the handle and paused — possibly to read the sign or to pull weakly before pushing the door open.
• 3 people pressed the button that automatically opens the door.

(I should add that vastly more people went through the doors as part of a group or just behind someone else. Counting them would’ve just skewed the results. I didn’t count any of the several dozen people who were near the door while it was already open or in the process of closing.)

Key Lessons

Respect standards and norms.

Most swinging doors on commercial buildings in this city swing out. We’re trained that when we approach a building from the outside, the door opens toward us—unless it revolves, slides, or otherwise opens automatically.

And then there’s the handle. The outer handle on these doors is designed to be pulled. It’s got a good grip. Good leverage. Handles designed for pushing are usually obvious about it—with a flat plate or a horizontal bar. Basically everything about these doors, except the little sign, tells us to pull.

“Innovate when you know you have a better idea (and everyone you show it to says “Wow!”), but take advantage of conventions when you don’t.” Steve Krug

Don’t expect people to read.

The “Push” sign barely helps. (I vaguely remember the sign being added after the building opened, and thinking, “I guess I’m not the only one who can’t use these doors.”)

The sign is largely useful for “error recovery.” It keeps us from giving up or breaking things. By the time we notice the sign we’ve started to struggle and feel frustration. It can even make people feel even more annoyed and ashamed. It might as well say, “Push, you dummy.”

Don’t make people think.

To complicate things even more, immediately beside these doors is a nearly-identical set that opens the other way. If you veer right, the doors open in. If you veer a few feet to the left, the doors open out.

Maybe this was done to make things easier for people. I can imagine a thought process that goes something like this:

• “Most people walk on the right.”
• “Pushing the door open is slightly easier than pulling.”
• “Ergo, let’s make it easier both ways by making it so the doors on the right are always pushed, whether entering or exiting.”

But logic often backfires when we’re talking about human behaviour. Opening a door isn’t hard for most people. What is hard is learning a new mental model (“bearing right through a ‘main entrance’ = push door”) and trying to remember where and when it applies (interior doors in the same building don’t use the same mental model).

“There’s great inertia in users’ mental models: stuff that people know well tends to stick, even when it’s not helpful. This alone is surely an argument for being conservative and not coming up with new interaction styles.” Jakob Nielsen

Different isn’t always better.

The worst that happens when you stick to conventions is you keep the thing boring. It’s probably ok to bore people for a few seconds while they walk through the door — or log onto your website, or open up your app, etc. The real value is what they see and do and feel and remember when they get in.

Things like doors and login processes are basic or must-have features that don’t increase overall satisfaction on their own. But they can decrease satisfaction when they’re missing or done poorly. This is where the Kano model comes in handy.

“Capabilities that users expect will frustrate those users when they don’t work. However, when they work well, they don’t delight those users. A basic expectation, at best, can reach a neutral satisfaction point where it, in essence, becomes invisible…” Jared Spool

Trying to improve basic features can be a dangerous game, usually with high risks and low rewards. Apple has played this game fairly well over the years, but usually for higher rewards, guided by long-term vision and strategy, not just immediate incremental improvements.

Even if the risks seem relatively low, people do have a tendency to notice and remember negative features more than positive ones. People remember feeling stupid. They remember being laughed at—or feeling laughed at.

That’s why you’re more likely to see a Facebook post calling your doors “the worst” (actual Facebook comment about these doors) than you are to see posts saying “those doors work great!” Especially when they don’t.

Change can be great, but count the stakes and make sure it’s actually an improvement. Make it worth the risk.

Originally published at https://res.im/blog/why-are-some-doors-so-difficult/

Plans have a purpose, but I’m always afraid of being too constrained by them in research, strategy and design work.

In academic research, plans help to ensure scientific rigor and secure funding, etc. In UX research, plans help manage expectations, coordinate teams, assure the bosses that we’re making good use of time and money, and generally maintain a degree of professional diligence.

But just as ‘no battle plan survives contact with the enemy,’ research plans have a way of changing once participants get involved. It’s not at all uncommon for interviews to go in an unexpected direction in the first few minutes, even when participants have been carefully screened. The uncertainty is both a bit scary and exhilarating — especially when it plays out in front of clients.

“Everybody has a plan until they get punched in the mouth.”

– Mike Tyson

During one session with a couple of continuing education students, it became clear that the participants were much more dependent on the print catalogue than we expected. Most of their comments revolved around making a 200-page PDF easier to find — despite also expressing frustration about it. I thought we’d be getting feedback about the website but they hardly used the site at all. Most of the prepared questions didn’t make any sense in that situation.

The participants’ fixation on finding the PDF and leaving the site as fast as possible felt like a classic case of “customers don’t know what they want.” The planned discussion was mostly a dead end, but we’d been given an unexpected gift: now we had an opportunity to observe people using the site somewhat naturally but in ways we hadn’t seen, based on real needs, to get deeper insight into the experience. So I turned my laptop around for the participants to use in some unplanned usability scenarios. The unexpected change gave us a better opportunity to see what people really saw and thought while on the site.

Uncertainty and change don’t just affect individual sessions; the whole process often needs to be open and adaptable to unexpected discoveries.

If you’re too rigid you often end up missing the best opportunities to learn. Over-planning reinforces biases and assumptions in a closed feedback loop that makes transformational insights harder to get. Rigid plans are often fine for evaluative research, like when you want to know how satisfied people are or whether they know where to click on an interface, but rigidity constricts the exploration and generation of new ideas. Transformative discovery is the art of serendipity — putting yourself in a position to be surprised by new questions.

“Chance favours the prepared mind.”

– Louis Pasteur

Planning helps anticipate some of the more predictable surprises. Planning forces you to visualize how an interview or group session or set of survey results might go, then tailor your questions, methods and materials accordingly.

A while back I started drafting a short questionnaire to recruit and screen participants for a personal project on how people listen to music. In the process of drafting the questionnaire I thought, “wait, what if someone says they ‘listen in the car’ but mainly to news and talk shows?” Podcasts and talk radio often do the same job that music does. “Would the question make sense? Would the answers be credible? And why should this study be limited to music?” It became clear that I needed to do some more open-ended exploration — talking to people — to better understand the scope of the challenge I wanted to address.

“Plans are worthless, but planning is everything.”

– Dwight D. Eisenhower

The plan itself is no guarantee that every contingency is covered, but the planning process helps anticipate problems and opportunities to learn. Planning makes us think carefully about the purpose and goals of our research. What decisions will we inform? What are the underlying questions we need to answer? Those answers then become the pole star we can use to navigate through uncharted terrain.

The most effective UX research plans, I think, adopt Agile and Lean principles.

Agile principles dictate that work should be iterative, collaborative, and open to change — with value delivered early and often. You might have a research roadmap (or loose plan) looking months ahead, but the plan is subject to change based on early findings and decisions. Instead of waiting to package everything in one big, hairy deliverable, insights are shared and applied as work progresses.

Lean principles go a step further and make the process more continuous. In a lean system, generally, work is pulled in small batches in response to demand. Lean research is a response to decisions that need to be made, questions that need to be answered, and assumptions that need to be tested — all of which continue to emerge during and after implementation.

The most adaptable research is a process of continuous discovery — not as an excuse for bad planning but as a natural response to evolving questions and needs.

Original post: https://res.im/blog/embracing-agility-in-user-research

Even as an admirer of the principles behind design thinking, I expected the hype to peak in the late ’00s. But instead of fading it’s actually finding a wider audience and pushing deeper into more industries and organizations. Harvard Business Review put it on the cover last September.

So what’s going on here?

First, I think what we’re seeing is not just some fad but a response to structural shifts in industries and organizations. At risk of oversimplifying it, the rise of design corresponds with the rise of software — especially software as a service — and the gradual and continuous evolution of processes and organizations around it.

Unlike hardware products, or even packaged software, software provided as a service over a network can be updated and delivered to users on the fly. Over the past decade, companies that figured out how to manage that continuous delivery process have had a competitive advantage over those that made customers wait six or eighteen months for improvements. By the time the latter company releases a new version of their product, the former company has already found new things to improve based on customer feedback.

Meanwhile, companies that weren’t initially in the software business have realized the value and importance of augmenting their products with software and other services. GE’s launch of a digital division is a prominent recent example. As a result, software (or “technology”) is no longer “an industry” in quite the same way that it was at the dawn of the PC era, so much as it’s a whole layer spreading out across virtually every other industry. Transportation, retail, hospitality, education, and entertainment have all been transformed (or “disrupted”) by software companies, and the incumbents have been forced to follow.

And all of that stuff needs to be designed — and I don’t just mean making surfaces pretty and usable but making sure these increasingly complex systems work together to serve increasingly nuanced needs. And those systems aren’t just the products and services themselves but the processes, teams, organizations, and business models around them. These need to be designed and redesigned as well.

We could quibble over what’s really “design” vs. what’s “architecture” vs. what’s “management” or whatever else, but any form of deliberate change is within the dictionary definition of design. Let’s not overthink it. More time and energy is lost arguing about the definition of design than will ever be lost misapplying it.

So now I look at it this way. What if management as a practice isn’t just temporarily borrowing from design, but some large part of it is actually becoming [or being displaced by] design? I don’t mean that everybody is a designer, but that a growing number of people in organizations are spending more of their time doing design activities. Calling a few of them designers is less weird or inaccurate than what most are already called, whether it’s something generic like manager, analyst or strategist, or something more contrived.

Management as a practice only really emerged after industrialized organizations demanded large numbers of managers simply to operate. Now we have another sort of industrial revolution that’s automating a lot of the work managers have traditionally done, like monitoring, reporting, coordinating, and even low-level decision-making. Needs are changing.

In the future we’ll spend a smaller proportion of time managing these systems and more time designing them. We might not call the people who do this “designers,” but in an important way design is slowly becoming the new management.

Before I Get Into Rdio…

Last week Sony announced the last-ever Betamax cassette will roll off the production line in March 2016 (which means someone, somewhere, has actually been using them all these years).

Legend has it that Betamax was better than VHS. The reality is a bit more complicated. Original Betamax tapes were only an hour long, i.e. not long enough to record or distribute a standard movie, which in hindsight is obviously a major shortcoming. VHS tapes had twice the capacity, partly due to their makers’ willingness to make them larger and more unwieldy. Most consumers didn’t care about unwieldy. Sony eventually extended their tapes’ capacity, but so did VHS makers. With VHS you could tape two movies, then eventually three or four. In 1993 you could tape two Blue Jays World Series games on one tape (or if you’re my mom, a whole week of your favourite soap).

The history of technology is full of stories like this. My elementary school had an underutilized Apple in 1985 and that was pretty much the last time I touched one until the mid ’00s. I believed that Apple computers were better in some ways that appealed to me but the uncertainty and cost of switching was always just high enough to keep most of us stuck with Windows.

A product’s own technical capabilities and quality aren’t all that matter when history decides whether it will prevail. It also takes timing, business savvy, good branding and strategy, and understanding what most people want (or guessing, unwisely). Sometimes it requires the ability and will to sell at a loss or throw money at it until attrition catches up with competitors.

We Know, But It Still Sucks

It hurts to be on the losing side, even if you’re only a consumer. Great products inspire love. They become part of people’s daily lives, part of our thought patterns and identities. Sometimes this is especially true for underdog products. Maybe it’s a kind of “Florence Nightingale effect.” We become more personally invested when we sense even a tiny bit of ourselves shaping a product’s survival or success. And for products we really love, our life stories can literally intertwine; we lose a little piece of ourselves when the product does.

It might seem like hyperbole but I thought about and felt some of these things this week when Rdio announced they’re shutting down.

Rdio might be my favourite service of any kind, and has been since I signed up five years ago. I looked forward to checking the new releases every Tuesday. Most of my favourite albums of the past five years I discovered through Rdio (a lot of them serendipitously via friends’ activity). Before I went on trips I always made sure my Rdio playlists and downloads were set up for the right soundtrack. Robinson Meyer’s eulogy for Rdio made me realize how good the community was on there.

People left comments on albums, and, lo and behold, the writing was good and interesting. Strangers constructed playlists that pulled from artists and albums you’d never heard of, but without the performative high/low-ness that afflicts so much online music talk.

It might seem weird to say, but I have a lot of warm memories associated with Rdio, in the same way that I look back nostalgically at discovering music in record stores, at concerts, and in friends’ physical collections.

Sadly, Rdio wasn’t for everybody. Outside of a small group of like-minded people, nobody I knew stuck with it (despite my best efforts). Meanwhile the same people who didn’t stick with Rdio raved about services like Songza and Spotify that emphasized the kinds of passively consumed, curated experiences that feel overmediated and just unpleasant too me.

Spotify seems to have improved in the direction I want it to but I definitely feel like I’m not the target Spotify user. There’s a bit of clumsy industry cruft to navigate around before finding the experience I’m looking for. (Picking a category before I browse feels unnatural. And after telling Spotify I’m a 37-year-old male and listening to one metal album, its top radio station recommendations for me are Drake, Justin Bieber, and One Direction?) Same thing more or less goes for Apple Music.

According to a story on it at The Verge, Rdio’s downfall can be partly attributed to too much focus on product details that too few people would appreciate, and not enough effort to market to the mass fixation on “free.”

Rdio was elegant and pure compared to everything else. I’d go as far as to say it was damn near perfect for a brief moment (notwithstanding the limited selection, especially in the early days). This is why Rdio inspired love and influence that went well past its modest mass market penetration, as Casey Newton pointed out in that Verge piece:

Among its subscribers were a small legion of user interface and user experience designers — one reason you see little touches of Rdio everywhere you look.

The things that product designers care about aren’t the same things that most consumers care about. Just as people embraced unwieldy VHS cassettes and endless Windows popups over more elegant alternatives, people embraced familiar music industry cruft for the sake of getting a good enough, free-for-most service. By the time Rdio started to respond, the outcome was already well in motion.

So What’s To Be Done?

But I don’t think the takeaway should be to cynically steer every product toward the lowest common denominator of consumer preferences. After all, Apple eventually thrived, as did Pixar, another company Steve Jobs nurtured through a formative phase in which high product standards put the business at risk. (We could add NeXT to the list and argue that without it, present-day Apple and the Internet would both be very different. That must count for something.)

There’s a whole list of “failed” products in recent years that we could say were “successful at being awesome,” which inspired some of what came next. Google Wave and FriendFeed are among the others that got me excited but didn’t win the popularity contest (I worry about Twitter ending up on this list soon).

Sometimes it can start to feel unrealistic to want to create something we’re proud of that’s also economically viable, like at some point there’s going to be a fork in the road and we can only pick one way: creative pride vs. commercial profit. A middle way is there, it’s just tougher to find; it requires comfort moving in either direction and the ability to switch at the right time.

And I keep in mind that companies that pick the cynical, profit-maximizing route at the expense of quality and love often fail too. They still need to do everything else right or risk coming away without even so much as a decent story to tell. And creating great stories is what it’s all about, isn’t it?

If instead we err the other direction and create something we’re proud of and it touches people’s lives and changes the future even just a little, well that’s definitely not nothing.