Comparing Amazon and Walmart with simple but critical usability testing tasks: finding and buying an iPad, who won?

Amazon and Walmart are kings of eCommerce. But how do they compare in usability? To answer this, I created a simple but useful usability test: something thousands of users were trying to do this holiday season, finding and buying an iPad.

The usability testing protocol I created was simple, but not meant to be exhaustive in terms of comparing the user experience of both sites. Rather, the test was a quick evaluation of how easy or difficult it was for users to find an iPad with the best possible features for the price (the value of which had to be less than $550) and then buy it.

Here’s the usability testing protocol I set up for the test. It’s simple, quick, but importantly meant to be directional only. I used usertesting.com as my tool for this test.

Amazon versus Walmart Usability Testing Protocol

Introduction: You are buying an iPad as a gift for a family member. You only have a total of $550. You want to buy the best one you can for the price in terms of functionality and features.

Task 1: Please show me how you would find an iPad or iPads that are equal to or less than your price range?

Task 2: Let’s assume you’ve decided to purchase one of the iPads, please show me what you would do to buy it. Please go through all the steps without actually purchasing it.

Tester Age: 18 to 65+

Tester Household Income: $40k to $150k+

Gender: Any

Web Expertise: Any

Country: United States

Number of Testers: Six total (3 for Amazon and a different 3 for Walmart)

Testing Dates: December 6-17, 2012

Usability Testing Results of Amazon versus Walmart

The results of the usability tests are revealing and point to several areas where both Walmart and Amazon may need to explore further usability optimization. And even though this test was simple, quick and used a rather small amount of testers (3 for Walmart and a separate 3 for Amazon), it clearly shows how even minor amounts of usability testing can reveal important places where the user experience can be potentially improved. For eCommerce, this also means improving revenue!

Let’s look first at the results for Amazon and Walmart in terms of how they performed for several key tasks, including:

1. Finding an iPad
2. Filtering product search results to find the price range that fits our tester’s budget
3. Being offered the opportunity to purchase a protection plan (something that no doubt is high on the Walmart and Amazon team’s radar as it is a good source of incremental revenue per shopping cart)
4. Purchasing the item

We also look at several other errors that seem to be obvious things that can be fixed, or at least evaluated.

And now, on with the results!

1. Amazon versus Walmart Usability Test Task, Finding an iPad

Winner Walmart

Amazon and Walmart take different approaches to displaying and filtering product search results. So a true apples to apples comparison is not possible. However, we can compare the overall ease of use of each system based on the task of asking a tester to “find an iPad in the $550 or less range,” a real world scenario.

Based on the results of this test, the advantage goes to Walmart. This is primarily due to the displayed list of results after the user enters iPad into the search tool. All our testers were able to easily navigate the results, and take the next step promptly, which was to use filtering to find the product in the right price range.

Amazon did not do as well in this test as it could have. The search results are critical to helping our testers to sift through the hundreds of thousands of products Amazon sells to find an iPad in the $550 or less range. Even here at the very start of searching there were potential usability issues.

One of our testers almost immediately became confused when he noticed that the top result for the search term “iPad” was an iPad 2 Second Generation, which caused him to spin off in a different direction spending larger amounts of time trying to find the newer models (as of the writing of this article the iPad Fourth Generation is the newest iPad). Interestingly, all testers mentioned that “typically the best product is at the top” even though this clearly was not the case, and all of our testers had to do a fair amount of searching by scrolling up and down, or clicking on various links, to find the newer iPad models that fit their $550 price limit.

I am guessing Amazon has a usability team so I’m hoping they can evaluate this test result, to determine if there’s a need to find a better way to put the newer (aka “hotter”) products at the top of their search results display. I’m thinking perhaps some Search algorithm testing is in order.

For Walmart, things went well for testers who used the search bar, but the one tester who did not use the test bar had a much harder time of finding iPads. Lesson for Walmart? Consider making your search bar bigger, to attract more attention and cause fewer users to try to navigate through a more difficult process.

Click to Play the Amazon Highlight Reel

Click to Play the Walmart Highlight Video

2. Amazon versus Walmart Usability Test Task, Finding an iPad Using Filters

Winner Walmart

A critical element of eCommerce is using filter tools to narrow search results, which both Amazon and Walmart do, but using vastly different methods. For Amazon, there’s not a specific filter tool that’s readily apparent such as with Walmart, however users do have the ability to filter results, IF they know where to look.

Interestingly, the testers using Amazon had a more difficult time finding the iPad that fit our parameters, in this case a model that gave the most performance and features at a cost of $550 or less, than did the Walmart testers. This was specifically because the Walmart filter tool enables users to easily filter based on price. Not that our testers found the Walmart tool without problems (which they did).

Still, Amazon’s filtering (or lack thereof) of product results based on pricing parameters was something that all our testers struggled with. All testers resorted to scrolling through pages of results, some gave up early and selected a product because it was listed near the top and seemed to fit the test parameters. In the real world, I’m betting this behavior happens more often than may be realized, I’m not sure always to the benefit of Amazon or Amazon users.

Several times, testers became lost in their search due to scrolling through so many results and had to “reset” themselves by going back to the starting results page. The inclusion of peripherals spread in what seems random fashion in the results did not help matters, as it made hunting in the results for the latest model iPad even more difficult.

Because of the extra cognitive load Amazon puts on users, we give the nod to Walmart for this part of the test.

Click to Play the Amazon Highlight Reel

It would be interesting to see what the usability test results for Amazon would be if they were to offer their users a filtering set of tools along the lines of the Walmart tool, versus what Amazon users currently have available.

Walmart Filtering Tools are Good, but not Great

Walmart has one advantage over Amazon in terms of our test of finding an iPad in our price range, and that is the filter tools on the left side. Interestingly, all of the testers used this tool, and all of them were able to reduce what was a much larger list of products down to those they felt met their parameters by using the tool. That’s not to say the tool didn’t cause issues. Several found the refresh that happens without warning rather disconcerting, and one mentioned that slides were preferred, as that way the exact pricing parameters they wanted could be entered.

Click to Play Walmart Filtering Highlight Reel

3. Amazon versus Walmart Usability Test Task, Product Protection Plan Offering

Winner Amazon

A critical element of eCommerce success is adding in additional SKUs to a shopping cart, in this case a protection plan. Typically this is good for the company, as it is an incremental source of revenue. But it can be a good idea for the shopper too. Reminding them to buy additional items or a protection plan they (if they are anything like my family) will end up using when something bad happens to their product is not a bad idea.

In terms of the offers, both Amazon and Walmart pop-up the protection plan, but that is where the similarities end. Notice the critical difference, Amazon has the “Add Coverage” button the bright, yellow, some would almost say Default button. Because of this, people evaluating adding the extra coverage may have more of a tendency to click the highlighted button, all other things being equal. In essence, the default is YES.

But with WalMart, note the choice is “I prefer not to add coverage.” Ouch. The default here is NO. Also, note that with Amazon you only have to click one button to make your selection, Walmart requires two clicks, one on the radio choice button, and then one way down at the bottom of the pop-up for “Continue.” My guess is WalMart is losing hundreds of thousands, maybe millions, of incremental dollars with their current protection plan offering user experience. Perhaps the WalMart usability or metrics team may disagree with me, but I would test a much more Amazon like user experience here, just to see if there’s a difference (I am betting lunch with the entire WalMart usability team that there is, if you know any of them forward them that message from me).

The Amazon pop up with the Protection Plan offer has a single button to buy the product

The Walmart pop up with the Protection Plan offer requires two clicks, and does not highlight the YES choice

And just to provide an additional data point, it’s interesting that the only tester to choose the protection plan was an Amazon tester, although there was a Walmart tester that was tempted.

Click to Play Amazon Protection Plan Highlight Reel

Click to Play Walmart Protection Plan Highlight Reel

4. Amazon versus Walmart Usability Test Task, Purchasing

Winner Amazon

In fact, both Amazon and Walmart are about equal in terms of the ease of moving through the buy-flow. Both have what can be described as best in class user experiences in terms of the shopping cart to purchase task flow. That said, Amazon has a slight edge with their ability to move users through the process with a bit less cognitive load, as witnessed by the several errors that occurred for our Walmart testers that did not happen for our Amazon testers.

Since so much went right for both purchase flows, let us focus on the errors we picked up, both in the buy-flow as well as in other places. Amazon more than once tripped our testers up with offers to buy a product at a price that seemed to disappear when they actually went to the results pages to find the product at that price. Walmart had several avoidable user errors in their buy-flow, mostly caused by simple things like not labeling required fields or hiding critical choices in the middle of a rather busy purchase page. Simple usability and A/B testing could easily improve all these easy to fix errors.

Click to Play the Highlight Reel

Summary of Amazon versus Walmart in Usability, Who Won?

So in summary, based on this simple usability test we performed, it would appear that Amazon and Walmart are about equal in terms of the usability of finding and purchasing an iPad, with Amazon winning two categories and Walmart winning two.

However, I actually believe that based on this test Walmart has the edge in usability. The primary reason? I believe Walmart provides an overall easier and faster user experience in the searching, filtering and vetting process associated with seeking out and purchasing a product.

The primary advantage Walmart has over Amazon is the availability of filters on the left side of the products search results pages. This filter set enables users to very easily target products that meet their parameters, to find the best product possible for the given budget range.

Dave Garr, co-founder of Usertesting.com

My friend Dave Garr is probably one of the most non-famous Famous usability and UX thought-leaders around. If you’ve never heard his name, you very well may have heard of his creation: UserTesting.com. As a co-founder of UserTesting.com way back in the dark days of 2007, Dave and his team have made a major impact for all usability practitioners; a fast, low-cost and useful usability testing service that provides results in a day versus what used to take weeks. When he’s not re-inventing usability testing, Dave loves writing and performing song parodies. While at Apple, he recorded these videos: “I Think We’re a Clone Now” and “Killing My Software with Windows”

If not busy enough completely re-creating remote unmoderated usability testing for the entire world, Dave won a Webby for his marriage proposal.

What’s your background?

My first brush with technology came during a summer job in college with a software company who developed EasyWriter, the first word processor for IBM’s PC. Fortunately, when I went through the interview process, no one asked me if I’d ever used a computer before.

I graduated from Cal Berkeley in Marketing, and I’ve overseen websites for several companies, such as Intuit, HP, and Apple.

I live in Palo Alto, California with my wife Elizabeth who encouraged me to pursue my startup dream, and my two young daughters who don’t seem to be even remotely interested in usability testing.

How did you get into the usability field?

I was managing Apple.com and I found myself drawn to watching Apple’s user experience labs. I was fascinated with how hard they worked to improve the out-of-box experience, or to decide on the use of color in the Mac OS, for example.

Then I read Steve Krug’s “Don’t Make Me Think.” It’s fantastic, particularly his chapter “Usability testing on 10 cents a day.” He makes it so simple: “Watch some people while they try to use [your site] and note where they run into trouble. Then fix it, and test it again.” That resonated with me. Since then, Steve has been kind enough to be a mentor to me.

What is it about your job that you most enjoy, or find most rewarding?

That’s easy. It’s when we release a new capability on UserTesting.com and a customer tweets or blogs about how it’s improved their life. Like one guy tweeted: “I nominate usertesting.com for a Nobel Peace Prize for preventing warfare between designers and developers. Don’t fight, test it and see.”

What is UserTesting.com and why should someone use it?

We like to think of it as usability testing without the hassle. You create the test, and we handle everything else, including getting the testers. We record the testers using your site, so you can virtually peek over their shoulders to discover your site’s problems. It’s $39 per tester and you get the results in about an hour.

Companies commonly test:

• Their own website and landing pages
• Competitors’ sites
• Semi-functional prototypes and staging sites
• Facebook games
• Mobile apps

What was your motivation for creating UserTesting.com?

I’ve spent an unhealthy amount of time in usability labs, and I was frustrated with how expensive and time-consuming it was. So I started doing a lot of quick and dirty usability testing with my family, coworkers, friends, neighbors, and — in those rare times that I could get up the guts–captive audiences at train stations.

So UserTesting.com solved a pain point that I had. And fortunately it’s a pain point that others have.

What have you learned while running UserTesting.com?

We launched our minimum viable product (MVP) four years ago. After every order, we emailed customers asking them “What can we improve?” We’ve received a lot of feedback because — believe me –our MVP had a lot of room for improvement. The request that we heard most often was this: “I want the participants to be my exact target market and not people on your panel.” So that’s the area we dedicated the most resources to build. Now participants don’t just have to come from our panel — they can also come from any of these places:

• Live visitors on your website who’ve been intercepted
• Your own customer list
• Participants from third-party panels

Another thing we’ve learned: website owners care a lot about their competitors’ websites. As Steve Krug says, “Someone has gone to the trouble of building a full-scale working prototype of a design approach to the same problems you’re trying to solve, and then they’ve left it lying around for you to use.” In particular, our clients want to know what their competitors are doing right, so they can “borrow” it.

What advice do you have for other start-ups that wish to create an online service, whether usability related or otherwise?

If you only take away one thing from this interview, then by far and away, my biggest recommendation is: come work for us! We’re growing and want to hire more people who are passionate about rescuing the world from hard to use products.

But if you insist on doing your own thing, then here are some thoughts…

Do what you love. Since I’ve had to immerse myself in the topic of usability testing for four years, it was good that I was very interested in that topic. It would be hard for me to spend so much time every day thinking about a topic that I wasn’t passionate about.

Work with people you love. I cannot overemphasize how important it is to work with someone that you really enjoy. My co-founder, Darrell Benatar, had previously been a friend and co-worker, so I knew how well we got along.

Love your customers, even if a few take advantage of you. I’ve learned that providing great customer support is crucial. We’re trying to mimic Zappos on this one. We’ll give customers a refund, no matter what. Even if they ran tests years ago, combined the best clips into a highlight reel, and shared that highlight reel with hundreds of people; if they ask for a refund, we give it. Our customers don’t abuse our return policy–less than 1% of our customers request a refund. Best of all, by having great phone, chat, and email support, we get tons of feedback from customers about how we can improve our product.

What’s in the future for UserTesting.com? What changes or improvements are you working on?

Okay, I’ll tell you…as long as you promise to keep it a secret (laughs).

Probably my hardest job is deciding which feature to add next. We’ve learned from ConversionRateExperts.com to rate each feature (on a scale of 1-10) according to “How easy is it to implement?” and “How important is it to customers?” We multiply these two figures together to give an estimated return on investment. Next we build the feature that has the highest estimated ROI and then A/B test it.

Our DNA is about listening to customers. When a lot of customers ask for the same thing, we usually do it. The biggest things they’re asking for now are:

• Improve our user testing of mobile devices. We’ve developed a mobile version of our platform, and making it better is a key focus of our product development for the foreseeable future.
• Expand globally beyond the US, Canada, and UK.
• UserTesting.com has made it easy for you to get feedback on your live website. However, as you know, the earlier in the development cycle you test, the easier it is to make changes. But we haven’t made it easy to test concepts. Shame on me for not doing a better job on that. So we’re going to try to make it easier to use UserTesting.com to get feedback in the ideation phase of the dev process.
• Consulting services for our enterprise customers. We create the test plan, make clips of the places where testers got stuck, and recommend how to fix the biggest problems found.

What do you think the next few years will bring for usability?

Computing is moving from one screen — a computer accessing the web — to four screens: a computer, a tablet, a phone, and a smart TV.

Let me quote a usability expert named Craig Tomlin! You’ve talked about the next big UX trend is understanding that “user experience” does not mean just “web site experience” or “mobile experience” or “phone experience” or “store experience. ”

Companies will stop designing each experience in a vacuum. They’ll start putting together the holistic understanding of the entire “user experience.”

You used the example of how someone buys a car. She may go to several websites to evaluate car brands. She may use her mobile phone to schedule test drives. She may ask friends on Facebook or other social sites about their opinions. She may build and configure her ideal car on her iPad. Eventually she goes to the car lot and negotiates with the dealer. Given that she has interactions that transcend any single experience, why would car companies design the user experience she has with the website without considering the other critical interactions she’ll have during the car buying process?

What’s next for you and your career in the next year or two, what would you like to focus on?

Sentiment analysis. Sentiment analysis is software that mines text for meaning and insight. It’s often used to extract opinions and emotions from social media to help companies determine how people feel about their products.

Sentiment analysis is an extremely difficult problem, but it’s a problem that will eventually be solved. When the problem is solved, it’ll cull through a massive amount of text and automatically call attention to the biggest, most frequently mentioned issues. This will increase the value of qualitative tools like open-ended survey questions and transcripts of user testing.

Thank you Dave!

My friend Mads Soegaard is a man possessed. Possessed by the feeling that information, especially information about usability, should be readily available to all. To that end he’s literally poured his full devotion, time and energy (and money!) into Interaction-Design.org. What would turn a mild-mannered programmer and all around good guy into a fanatical devotee of web-based education for all? Well, read on!

Q1. What’s your background? Where did you go to school, what subjects interested you?

Mads Soegaard, founder of Interaction-Design.org

I’ve taken the majority of my education here in Denmark. I have a master’s in “Information Studies” and studied a PhD in computer science, which I never finished.  I love anything that’s highly theoretical and highly practical at the same time. I get bored when I do too much of one or the other: After going to an academic conference and reading a lot of academic papers, my fingers begin to itch and I want to do practical programming and design wireframes. But after a while I begin to long for the theoretical stuff again.

Q2. How did you get into interaction design and usability field?

It seems I’ve always been here. I started a web development company way back in 1997 and our quickly made usability our hallmark. Since then, it has stuck with me. I think it’s because I have a love-hate relationship to technology. Some days I really hate technology, swear at it, and find it so frustrating, time-consuming and nerdy. On other days, I love it and appreciate how much it helps me.

Q3. What is it about interaction design and usability that you most enjoy, or find most rewarding?

Making cool stuff that works well and makes other people happy. It’s really as simple as that.

Q4. You founded the interaction-design.org site.  What is interaction-design.org and why should someone use it?

It’s a new model for publishing. Instead of our community producing books that are read by only a few thousand people in the Western intellectual hemisphere, we want to teach millions of people about usability and interaction design. Not just the paying ones but everybody. From New York to New Delhi.

Q5. As founder of interaction-design.org, what was your motivation for creating this tool – why did you believe interaction-design was needed?

Because I‘ve read so many fantastic books by fantastic authors. I believe these authors have the minds to change the world. However, they are not going to – as long as their works are only read by a few thousand people. We need top-grade educational materials to be accessible online for the whole world to enjoy and learn.

Q6. From your experiences with how people use your website, what content or key learnings seem to be the most popular, and why do you think that is?

The encyclopedia and our calendar are the most popular sections. They get about 3,000 unique readers a day.

Q7. What advice do you have for other start-ups that wish to create an online education service, whether interaction-design related or otherwise?

My advice is to:

• Save up a lot of money – and be willing to spend it all during your startup process
• Be able to live on oatmeal for several years
• Be willing to sell your car, remortgage your house, and let go of your fancy title
• Have a determination like few other people
• Put almost everything on the line

Q8. What’s in the future for interaction-design.org, what changes or improvements are you working on?

Well…that’s a secret but it will be revealed August 1^st, I hope. If we’re not late J

Q9. What do you think the next year to two years will bring for interaction design and usability?  Do you see it growing, if so by how much?

I see it growing a lot. Our everyday lives are getting packed with technology and every little shop on every street corner has a website. So whether you’re designing websites or mobile phones or household object, you’ll be needed more and more in the coming years.  People get frustrated with technology every day and we’re like doctors in that sense. We cure frustrating and time-consuming technology.

Q10. What’s next for you and your career in the next year or two, what would you like to focus on?

Well…that’s a secret but I’ll be happy to share the news with you on August 1^st

Thank you Mads!