• Why are they coming?
• Where are they going?
• Where are they coming from?
• What are they doing?

From the point of view of the organization, questions cover:

• Why do we care?
• What are our goals?
• Is it working?
• What should we do?

The problem and the challenge is that Web analytics gives you an ocean of data.  Frankly it can be overwhelming. The data is rich and deep, and only a click away. But what does it mean?

The Web Analytics Activity Cycle

In an ideal world the Web analytics activity cycle has four parts.

Discovery. First, its all about discovery. In other words, what is the business/Web mission, purpose and/or goals? How mature is your Web analytics team – the who, what and how about your program. A good discovery session also explores the organization’s pain points and challenges – is there agreement on priorities?

Translate Goals into KPIs. After you’ve gotten your team on the same page about these issues, its time to translate your Web goals into Key Performance Indicators, or KPIs as they are commonly called in the industry. I’ll discuss some of my favorites deeper into this post. There are some interesting resources out there to see what others use, including the KPI Library.

Setup and Configure Tools. With your KPI picks in place, then it is time to setup and configure your tools. A free place to start is Google Analytics, paired with a heat-map tool called Crazy Egg. If you are feeling green, Google Analytics offers some good online training. There is also a good screencast describing how to use Google Analytics produced by Beth Kantor that I recommend. To meet vendors and practitioners in the Web Analytics space, consider joining the Web Analytics Association which has begun offering a Web Analyst certification exam and offers regular Webcasts for members.

As a Web analytics manager be prepared to offer some training to those on your team about the basics to set expectations. Also be sure to get an historical baseline so you can compare your growth to something real. When you configure your tool, make decisions about whether you are going to count internal traffic or configure the analytics platform to exclude internal visits and just count external customers.

There are a lot of handy tools for Website analysis out there, some of which are free and others that charge thousands of dollars a month for very rich data streams. What has been a real game changer, however, is Google Analytics, formerly known as Urchin before it was acquired by Google. When Google released the free version to help create a class of informed customers who are (hopefully from Google’s perspective) purchasing ads on the search engine, it unleashed the ability for any Web site owner to see an incredibly deep amount of behavioral data about online visitors. The Google Analytics dashboard lets you see site usage, a geographic map overlay, a content overview and traffic sources among other things. In fact, the map lets you click into a state and even view which cities are sending visitors to your pages.

Analysis. The final and most important part of the activity cycle is the analysis. Simply sending a spreadsheet or dashboard link won’t do if you are serious about KPIs. Instead, pay careful attention to trending data — current versus baseline data. Establish your insights from the data, weigh the impact of making changes and manage for quick wins and long-term focus alike.

Adopt an Action Orientation

Before we go much further, its important to address the elephant in the room. You must recognize and grapple with analysis problems. What are you measuring? Is it subject to misinterpretation? Or do you struggle with a hurricane of demand for data that swamps your ability to do analysis to begin with?

You may also suffer from some management headaches. There may be very low engagement by your executives. It could be that your rhythm is to provide steady reports by email and there is simply never any action on the data. Missed opportunities can be legendary and frustrating. And if you are ultimately in charge of Web analytics, you will have to set priorities for your staff’s time and resources because nothing is ever available in an unlimited supply, in spite of good intentions to deliver the gold.

If you run your Web analytics work out of your marketing department, chances are your job is very complex. A McKinsey study shows how these marketing manager/executives spend their time on a variety of Web related activities. “An explosion of customer segments, products, media vehicles and distribution channels have made marketing more complex, more costly and less effective,” the authors wrote. The time study shows the percent of your day on Web related activities includes:

• 15% – on Web development and maintenance work
• 10% – on search engine marketing
• 5 % – on search engine optimization
• 5% – on email marketing and direct mail

If you elect to focus your limited attention span on Web analytics, rewards await.

You gain deeper understanding of your site visitors. You have the opportunity to improve user experience. You get the opportunity to make smart decisions and improve resource allocation. And you get the ticket to a greater return on your Web investment. These rewards come only if you have adopted an action orientation.

An action orientation for your KPI program starts with developing your selection criteria. The KPIs must be relevant, timely and instantly useful. Resist the urge to showcase everything and start small. For example, just five data-points — traffic/reach, search, satisfaction, actions and outcomes — can be a satisfying beginning to your work. Once you have the KPIs locked in, the next critical step is scheduling reports. I recommend not less than quarterly. Reports should also keep the audience in mind and be edited for their particular needs and attention span. Finally, do not neglect meetings. Sending out an email link will not do if you want to a reputation for action.

The action orientation follows a cycle of its own — identify the problem, the metric, your hypothesis, fix the problem, test the fix, analyze the results and take action to modify the fix as needed. Then the cycle starts anew.

Make Those KPI Reports Audience-Centric

Making your KPI reports audience centric is one of the keys to success. At the AF Portal, for example, we identified three key groups who needed regular KPI reports.

The first group – senior leadership – received a culled down executive size report of no more than three metrics. The idea was to have everyone on the same page with something that was easily digested and summarized and could fit on an index card in a pocket if desired. For our executive KPIs we selected reach, search and satisfaction.

The second group were content managers for the 20 Major Commands at the USAF. These team members had responsibility over the growth and maintenance of the AF Portal and its adoption at the headquarters command and the various Air Force Bases at that command, including the dissemination of policy, adoption and all the other day-to-day operational details. For this group, we identified seven KPIs that mattered, which they would get on a regular basis. These KPIs included the first three the executives got, plus four more: top entry (to capture popularity), feedback, page influence (to get at top editorials/stories), and savings (to monetize the top ten downloaded documents, which helped the team get their arms around return on investment and changing the business of publishing content).

The third group was the AF Portal team, responsible for the day-to-day operations of the enterprise effort. Here, team members got all seven metrics that the Content Managers got, along with three more: adoption (to capture registered users), loyalty (to get at time per page and trends) and help desk (where we pinpointed the metric to problem types only, although other help desk metrics were circulated to another team more focused on that part of the effort).

In identifying the KPIs that you will share across your organization, it is vital to link it to strategic questions that get at the overall goals and challenges your organization faces and answer the burning questions you have about whether you are effective. For example, a set of conceptual KPIs for one organization included the following five:

• Reach: does our audience know about us.
• Relevance: are we providing what our key audiences want?
• Packaging: is the information we provide in a consistent and usable format?
• Access and Collaboration: are staff experts made available and used?
• Quality: are we delivering a superlative Web experience?

Leap into the Future

If you are in charge of your organization’s Web analytics program, and you want to leap into the future, you must negotiate agreements and hold routine voice-of-the-customer meetings. The agreements should cover the format of your reports, who owns which metric (spread the love!) and agreement on the audience segments who will receive your metrics reports. The meetings will be critical to your success. Do not fall into the rut of pressing the send button and thinking your job is done. Hold meetings not less than quarterly with all appropriate stakeholders invited.

One final part of your job which will impact your success is how you choose to present your data. Style matters.

It is important to include trending, so users of the data can compare results over time. The reports goals should be to make the data actionable, hold owners accountable, improve take-aways and seed discussions that will help you and the organization reach decisions about how to manage your Web site. I have found the best format to reach these goals is a quad-chart style dashboard. This is a tip I picked up at a Web analytics conference from a presentation led by Avinash Kaushik, then with Intuit and now with Google.

Avinash’s model dashboard is easily created within a powerpoint slide. The headline is the name of your metric. At the top, you include a red-yellow-green dot to indicate the health of this metric, along with the name of the person in your organization responsible for managing the health of this part of your Web operation. The quad chart itself includes these four parts: in the upper left you present the data in some sort of chart format ideally. In the upper right corner you explain in plain English what that data is telling you – your insights. In the lower left corner you write down your bullets for what you are recommending as actions based on the data’s story. And finally in the lower right corner you remind the team what actions you SAID you would do at the last meeting and what the status is of where you are on those changes and action items.

I like this dashboard format a lot. It simplifies the data into a complete story-board that helps the team talk about it and take corrective steps if necessary. It holds the owners feet to the fire – nobody wants to be a red-dot for long. And it helps everyone see clearly what the to-do list is (or can be), new and old.

Put Your KPI Quad-Chart Into Action

Putting this into action at the AF Portal, we started with surfacing the key questions we wanted to use the data to measure.

• Are we paying attention to the right content? Which top level navigation button gets the most clicks?
• What is the status on migrating applications (a key business driver for the AF Portal – that applications move to a common platform for which the AF Portal was the presentation layer)? Are we focused on applications our users care about?
• Are the complaints about search justified?
• Of all the new functionality we are introducing, which features are getting used and where must we focus our outreach efforts?

Our KPI on traffic and reach answered question one. We showed our team the top six tabs using unique visitor data and page view data, in raw numbers and percentile format. The results showed us that although most of the team’s effort was spent on building out the organization pages, that the top visits to the AF Portal were on the customizable workspace pages followed by the LIFE pages, which were managed at the enterprise level. The first time this quad chart debuted, the action items were about seeding discussion on how the absence of this type of information left the team “flying blind” in managing its resources and expectations for growth.

A second KPI on traffic, focused on applications, helped us address our question about migrating applications. By clustering the applications into master categories by application type – self-service, mission-based, training or content related – we were able to shine a light on the success of the drive toward self-service applications as a replacement for many walk-in, phone-in services at the USAF. A need to examine the way the feedback loop operated with these self-service applications was also brought into focus by the data showing their popularity. With regard to the training application group, the need to simplify and unify how users found these tools was an active discussion item which emerged from this presentation of the data.

Looking at the search KPI was also instructive. Because it was a distributed publishing model at the AF Portal, we were able to isolate the business unit which owned the content which was showing up on the majority of the searches and work with them to optimize search. The results validated that search quality was a valid complaint and spurred action to provide better tips and training to the content manager community.

Our metric on adoption showed the raw tally of how many users were subscribed to various features that had been added to the AF Portal platform, some with inconsistent marketing and promotion and others with great fanfare. The need to do more consistent and full outreach on these capabilities came into focus. In addition, it sparked a discussion on tracking measurable goals for adoption and the need for a communications plan. One diamond in the rough that we saw was not adequately embraced at the time was Really Simple Syndication, which was relatively new at the time.

Summary

All of these quad-chart style dashboard slides are shown on my slideshare summary on KPIs which inspired this post.

What Web Analytics managers are confronted with when it comes to standing up a KPI program is an ocean of data that is easy to drown in. A well-constructed KPI effort can be your lifesaver. If you haven’t started yet, the question is: what KPI program are you ready for? Hopefully this write-up has inspired you to get started or to add new tricks to your current practices.

The Seven Types of Waste

The 7 Types of Waste are:

1. Defects
2. Overproduction
3. Excess Inventory
4. Motion
5. Processing
6. Transportation
7. Waiting

Here’s how we dissected the help desk issues using the framework…

Defects. The defects that existed with the help desk at the time had a myriad of root causes which were systemic across several business units who shared responsibility for this operation.  These included: the questions and answers,  bad instructions, poor training, bad internal communications and an overall lack of standards.  The result of not dealing with these defects was that we had upset our customers, wasted consumer resources and cost the USAF money.

Overproduction and excess inventory were not issues that could be applied to the issue at hand.

Motion. The problems with motion included several root causes.  There was inefficient flow of material, inefficient procedures and an overall lack of standard work practices.  The result of this was labor cycle time was wasted and poor labor efficiency existed.

Processing. There were a number of root causes related to processing.  Processing problems were rooted in non value-added work.  Also, there was a lack of attention to making changes in what was needed – a problem that could have been helped by asking folks to routinely ask “why” five times before moving ahead.  Customer needs were also not identified.  Variations existed in processing as well, due to a lack of standards.  Operator error was also evident.  The result of these processing issues was that the system suffered from delay, processes that don’t add value and an increased opportunity for error.

Transportation. Although it is not immediately intuitive that transportation issues would apply to a help desk team, we found they did.  Root causes of transportation issues included inefficient facility layout, a lack of flow, non value-added operations, long set-up times and a “batch” mentality which led resolution to be done only when a requisite number of issues had accumulated.  As a result of these issues, additional time was needed to resolve problems, space was wasted, there was increased opportunity for damage during handling and there were also equipment needs to review.

Waiting. Waiting was the root cause of all other wastes. There was insufficient capability, non value-added processing, non-standard work and poor material flow.  The result of waiting was increased lead-time, increased work in process and a response to the customer which is slowed.

Breaking it Down

Breaking down a problem to component parts and focusing on the assembly-line style parts, helped the team reflect on priorities for improvement. By translating the root causes and pin-pointing the results of the problem for the organization’s customers we also put a human face on the problem to spur urgency needed to fix the issues.

I’ve always believed that once you describe the problem and forge agreement with all parties on the description – you own the solution. This system helped us have a common language on describing the problem. The solutions that emerged on our punch-list for priorities for improvement set the stage for a turn-around to the relief of many help desk customers.

In spite of Toyota’s troubles, I think the legacy of the “Seven Types of Deadly Waste” will continue to inform management decision-making for those involved in business process improvement teams. Can you see the framework being applied anywhere in your organization?

Governing Processes

Manage the Program

1. A clear structure for monitoring operations
   • Web Governance Defined
   • My Web Dream Team
   • How to Measure Leadership
2. Policy and Procedures
   • Policy Adoption
   • Lifecycle Management for Policy and Documentation
   • Emergency Controls

Plan and Execute

1. Design Strategy Step-by-Step
2. Planning | Congress Online: A Cautionary Tale
3. Understand Your Audience
   • 80 Percent Lurk
   • Nine Tribes of the Internet

Measure – Monitor Program as a Routine Business

1. Customer Satisfaction | All About Customer Satisfaction
2. Heuristic | Why Web Scorecards Rock
3. Usability Testing | Usability Testing: Options to Fit Any Budget
4. Comparison Benchmarks and KPIs | The Value of Key Performance Indicators: Unlock Your Data
5. Regular Audits | Five Magic Interview Questions
6. Improve Services | Review Your Team’s Performance

Core Processes

Provide Valuable Content

1. Confidence
   • Ease of Use
   • Readable
   • Look and Feel
   • Link Behavior
2. Transparency | The Next Administration and the Internet
3. Trust
4. Quality: Improve Content | The Need to Weed

Help People Find Information

1. Navigation | It’s All About Navigation
2. Search | My Sweet 16 SEO Tips
3. Efficiency
4. Marketing
   • I Heart Widgets
   • How to Work with Bloggers
   • 12 Steps to Writing Better Email
5. Access and Sharing | 40 Places to Find Social Media for Your Web Site

Do the Feedback Loop

1. User Help | Actionable Instructions
2. Workflow
3. Communications
   • Message Box
   • Ace Your Presentation
   • Meetings | Why Meetings Fail
   • Unified Info Hub
   • Social Media at the Office
   • The Power of Story-Telling

Enabling Processes

Care for People

1. Staffing
2. Training | Web Training Checklist
3. Content Help
4. Recognition, Rewards, Sanctions

IT Support

1. Publishing Tool
2. Requirements Gathering | Crowd-Sourcing Your Requirements
3. Specifications on Requirements

Provide Infrastructure

1. Site Performance
2. Platform Hosting| How to Pick a Web Host: 8 Factors
3. Domain Administration

Resource Management

1. Control Spending | Go Paperless!
2. Enterprise Architecture
3. Budget
4. Acquisition | Model Web RFP

The Benefits of RSS

When Web content isn’t trapped inside a specific Web site, it can be consumed just about anywhere. Many elect to use news aggregation services like Google News. But if you are into online organizing, you also want to leverage getting your content used freely in other websites so you can get as many eyeballs on it as possible. That makes marketing and sharing Web content a whole new ballgame.

In a post about 10 Essential Social Media Tools for the Office, I pointed to a great summary “ode to RSS” by a well-known tech blogger to highlight the benefits of hopping on the RSS bandwagon:

“A wonderful (if dated rant) by one of the Blogospheres most prolific consumers of Tech Blogs – Scobleizer – written in 2004 identifies the benefits of using Really Simple Syndication services to consume your online news makes sense: makes you more productive by serving up only the new stuff, no more wasted browser loading time, no advertising clutter and easier reading in same font.”

“But My Site Isn’t Built in XML!”

So you may be sold on the benefits of RSS, but the hurdle to making it happen is not desire but technical know-how.

What happens if you are in charge of a Web site that is not built in the XML format – the essential program language to enable an RSS to work? Is your only option to ditch the old format and painfully migrate content over? Just thinking about the broken links from URL extensions that end in something other than xml (like .htm, .html, .asp, .php) leaves my head hurting. That was my situation with my corporate site when I walked into a Netroots camp session led by online organizer extraordinaire Michael Whitney.

Along with a band of other RSS wanna-bes, we huddled in a room with Michael as he explained how to “hack” an RSS feed without going through the pain of building an XML site.

Use Delicious Bookmarks as an RSS Work-Around

The secret is leveraging the Delicious Bookmark service.

You see, Delicious lets you save and “tag” bookmarks from all over the Web on the fly, labeling them with names and keywords that are meaningful to you so you can go back and find the content later. What’s more, Delicious lets you save things online, so they are not trapped on your desktop and can be shared with others. When you save a bookmark to a Delicious account, you can title the content anyway you please and add up to 1000 characters and give it as many tags as you please.

And here’s the bonus: each and every tag you create becomes its own separate Web page that is automatically an RSS feed! That’s right, automatic RSS-feed ready page, with no special coding required.

As I wrote in I Heart Widgets this little trick helped me create an RSS feed of my articles on my html corporate site. All I had to do was create a tag – I selected “buzz” the name of my newsletter – and then save each article into that tag in my Delicious bookmarks. I added a paragraph or two summarizing the article to beef it up, and my newsletter was suddenly and magically recreated as an RSS feed. You can see what I did by checking out http://delicious.com/KMcShea/buzz

Some Extra Steps

There are a few extra steps you can take to package up your RSS feed to make it easier to consume and manage for you and your readers alike.

First, get a feedburner account from Google. This will let you enjoy some cool services, including a terrific graphic user interface when you sign up for a feed that’s very browser friendly, good analytics to measure your success, and even a way to let your subscribers sign up for an email when the feed is updated.

The other thing you should do involves a wee bit of coding on your html or non-xml site. If you want the RSS icon to be in the browser search bar automatically, which I’m convinced is where most people find it, you need to add a few lines of easy code to the header tag of your template and/or Web pages. This tutorial on RSS Auto-Discovery explains what code to use and how to implement it.

Summary

I think RSS Feeds are a game-changer for Web sites. Imagine a world where you created a feed, hacking one with Delicious if you needed to, and then started some old-fashioned marketing with affiliate Web site owners. Tell them you can bring them up-to-date news on a topic they care about, and to serve their readers all they have to do is “set it and forget it” with this little RSS feed.

Website owners also love RSS feeds because it builds instant traffic incredibly quickly when implemented and marketed well. When I got to visit the Fox News offices in NYC a couple years ago, their Web people for their business site were chirping about how using syndication (another word for RSS feeds) had helped them easily lap their CNBC competitor in just a few months in traffic visiting their site.

With these tips and tricks on how to get an RSS feed stood up without any XML coding, your excuses are over. It’s time to get on the RSS bandwagon.

I’ve cross-posted a word version of the Web RFP Template at my Squibd site and the html version is found here.

When I started my business back in 2001, one of the first articles I posted was this Web RFP Template freebie. (I’ve cross-posted all my business articles and resources to this blog.) I’m kinda tickled to say it has been hanging onto the number one Google search result for “Web RFP Template” for a good while now.  Lately it regularly brings my business site 600 – 800 visitors a month.

I hope this best practice offering helps make your Web development acquisition duties a little less reinventing-the-wheel and a whole lot more turn-key. Enjoy!

DESCRIPTION, PURPOSE AND OBJECTIVES OF WEB SITE

1. Describe the Web project you wish to undertake.
2. What problems will this technology project solve or what new capabilities will it provide for your organization?
3. If you have made a decision beforehand, indicate whether this site relies on a database to display information or will be “static” HTML

BUDGET

1. Provide a budget range so your solutions providers can scale their proposal accurately
2. Indicate whether you organization is ready to start work, or you have to raise money to begin the project. If you do need to raise money, vendors may help you describe the project in order to maximize your chances of receiving funding

TERMS AND CONDITIONS

1. Organization must own, have full access to and have the right to customize site code.
2. Terms for proposal:

• Relevant dates (see time-line item below)
• Proposals should be delivered to {name}. Copies should be {postmarked or received} by {date, time}
• Please provide {#} copies of any proposals submitted
• All proposals must include a statement of authorization to bid signed by a principal of the responding company
• All proposals must use the proposal format outlined in this RFP
• Parties submitting separate proposals may not discuss pricing information or they will be ineligible to bid on the project
• Bidder status: bidder must disclose any relevant conflicts of interest and/or pending lawsuits

BACKGROUND OF ORGANIZATION

1. Tell vendors about your organization. Provide links to information on your existing Web site, if applicable.

• When was your organization founded?
• How many staff members does your organization have?
• Optional: What is your annual operating budget?

AUDIENCE

1. Define the audience that will be using the site. Do you have a defined group of users or “stakeholders” who will be regularly using the site and have different needs?
2. What are the top tasks of each audience type?
3. Will the entire site be accessible to the general public?
4. Do you need the capability for staff or members to “log in” to access special information?

TOOLS AND FUNCTIONALITY

1. Essential components: are there certain features that are essential to any proposal you will consider?
2. Non-essential components: Are there other features of your site that you would like but are not essential to submitted proposals?

REPORTING NEEDS

1. If the site is database driven, what types of information will you regularly need to draw from it?
2. Will you need to integrate any Web analytics tools such as Google Analytics?

SITE SPECIFICATIONS

1. Design parameters
2. Accessibility/Usability

• Who certified section 508 compliance? Which factors and how deep?
• Usability testing baseline, mid-stream (if budget permits) and beta before launch will validate navigation choices and measure your success

3. Platform, if applicable

• Is this a redesign of a current site? If so, in what language is the current site built? (ex: cold fusion, ASP, PHP, Dreamweaver)
• On what format is the current site hosted? (ex: NT, UNIX, Apache)

4. e-Commerce

• Are you using an e-Commerce system now?
• If so, what are you using?

AVAILABLE TECHNOLOGY RESOURCES/INTEGRATION ISSUES

1. Do you have existing pages of Web content that you expect your Web developer to add to the new site? (Provide a map of your current site indicating how many pages will need to be transferred)
2. Do you have existing databases that will need to be imported or connected to the new site?
3. Do you have existing e-Commerce systems, web forums or other tools that the new site should be connected to?

STAFF RESOURCES

1. Point of contact/ project manager
2. Do you have a “Web team” of technical and non-technical staff members that will be reviewing proposals? If not, consider forming one.
3. Do you have a technology staff that vendors should plan to work with?

PROPOSED TIME-LINE

(Please indicate the time-line you expect vendor to adhere to)

1. Develop RFP process (forward email address for questions on RFP)
2. RFP release date
3. Submission of questions on RFP
4. Notification of Intention to bid
5. Answers to questions emailed to all bidders
6. Proposals due
7. Finalist interviews
8. Proposal award date
9. Initial meetings
10. Quarantine begins (no new concepts/functionality added)
11. Beta site
12. Proposed site launch

FORMAT FOR PROPOSALS

1. Executive Summary
2. Technical Volume

• Web development process: explain the process you will follow to build the Web site, including major milestones and evaluation
• Address usability standards and testing
• Address any important technology information and specifications used in your solution (languages, platform, etc.)

3. Management Volume

• Organizational structure: communication process; including lines of reporting and any special tools used
• Schedule of deliverables; include major milestones and testing proposal

4. Budget Volume

• Break down cost by production hours, tools and functionality (See tools and functionality section above)
• Maintenance and support: ID any costs that should be assumed as part of the site and ongoing costs for maintenance and support we need in the future
• License fees: ID the costs we will need to pay to develop or host the site
• Hosting: ID whether we must or are highly encouraged to host with your company. If hosting is provided as an option or requirement, provide pricing options
• Training and Style Guide: ID costs to train our staff to use site tools and provide a style guide
• Other charge areas: Please ID whether there will be other expenses, consulting fees, future work, etc. to complete this project

5. Attachments

• Qualifications and Experience: relevant case histories with information on accessing online demos or examples
• Biographies of all who will work on account
• Professional references

NOTE:

This resource was originally published at http://www.emeraldstrategies.net/web/WebRFPTemplate.htm In the years for which Google Analytics data is available, between April 1, 2008 and April 1, 2010, this page has received 13,170 page views and 11,062 unique visitors.

[What follows is a script from the Slidecast on this topic found above. It has been slightly modified to remove references more appropriate to a presentation accompanied by slides]

Usability: an Essential Part of Great Web Sites

If you’ve been thinking lately that it might be time to start some usability tests on your Web site, there is a lot of things to consider. This presentation reviews three popular options that can fit any budget and gives you all the basics to get started.

Usability is an essential part of all great web sites. It is the intersection of where your content meets your technology. From a users’ perspective you are answering questions

• Are you happy or are you frustrated?
• Are you being productive or are you wasting time?

From a usability engineers’ perspective there are a variety of tools and techniques and processes to measure the answers to these questions.

User Satisfaction

Now when we talk about satisfaction and loyalty, there are really two tiers.

At the first tier, rational loyalty answers three questions.

1. Are you satisfied
2. Would you recommend it to others and
3. Would you come back?

At the higher level tier we’re talking emotional loyalty.  And here you are striking cords about confidence, integrity, pride and even passion in your product or service.

Before we go much further, we should probably acknowledge the elephant in the room.  The fact of the matter is, whether your site practices good or poor usability has an impact on visitors.

If a visitor encounters a site that is badly designed, they are going to leave. If it takes too many clicks, they are going to leave. If they are in the middle of a shopping experience or other transaction, they are going to leave.

If they don’t like what you have to offer because it is just too hard, they are going to leave. Organizations are losing because of this.

And yet organizations aren’t even adopting the simplest practices.

Why Usability Works

You are viewing this post because you know that’s a problem. What you need is the ammunition to sell usability inside your organization.

Here’s why usability works.

• It ensures that your site is going to be easy to learn
• It helps your target audience get things done
• It is going to help your users find your information, perform tasks, understand the content and accomplish their goals.  And do so without wasting time
• And it is going to bring you satisfied users who like what they see, recommend it to others and decide to return

The bottom line is you can bowl a strike when using usability.

• You get actionable results
• You don’t reinvent the wheel
• And when making changes and leveraging resources, you get to lead with facts, not opinion

When you commit to usability that means you are putting the interests of your users front and center.  Who are they? What are they like?  When do they visit your site? What do they want to achieve when they come to your site?  Fundamentally you need to understand the answers to these questions, and usability can show you how.

What’s the Right Test for Me?

There are many ways to find out which tasks your Web site visitors consider the most important and which tasks are the most popular.

You can gather research from interviews, examine your call center data, conduct surveys or field studies and even mine your search log analysis.

But you also have to make sure that you are strategic.  That means you have to assess the goals of your business and your organization.

Here you want to talk to the leadership in your organization as well as your stakeholders.  You should also look at the relevant documentation and look at your competition.

So let’s suppose I’ve sold you on the idea of doing usability testing.  You are probably wondering, what’s the right test for me?

Well you have four choices. In combination or as single shot projects they all have pluses and minuses.

1. An inspection evaluation is often called an expert review and gives you a scorecard.
2. Human performance testing has actual users clicking through your site trying to perform top tasks.
3. Software inspection is out there, we can use lift machine by usable.net to supplement our expert reviews and be another set of eyes on the data.
4. A more full throttled approach where you do card sorting and audience feedback to really get a handle on how your navigation is performing.

The Expert Review

Let’s dive into the details.  I’m going to start with expert review.

The deliverable here is a scorecard.  What i really like about the scorecard that I’ve developed is that it follows this value ladder. The idea that users start out at the bottom rung and they need to be aware before they are satisfied, they need to be satisfied before they are confident, they need to be confident before they trust and they need to be trusting before they are loyal.

Because we map back all of our indicators on the scorecard to these metrics you have a step-by-step list that prioritizes what to work on first.

Our scorecard helps you climb the ladder, so in step one it is all about awareness.  What is the page weight like? How about accessibility and marketing your site?

Next, it’s satisfaction.  Are you delivering an experience that is free from errors? If you do make a mistake, can you recover quickly?  So what is the help section like? How about those broken links?  What about search and indexes?

At a higher level you are looking at issues of confidence.

Is the site learnable? Can you find what you are looking for?  Here we look for measures of navigation, link behavior, readability and the like.

When we reach even higher on trust issues we look at privacy policy, we look at content in the about us section. We look at whether you have a review schedule for your content and are you avoiding duplication.  And do your pictures load right?  Have you optimized them for the Web?

You Get a Web Scorecard

At the end of this evaluation you get a scorecard.  Your scorecard is a roadmap to help you identify where you need to tweak, where you need to overhaul and where you are doing well. We’ve completed over 200 web site evaluations for the US House of Representatives.  Here we had a custom scorecard.

The current scorecard instrument that we’re using is a little more streamlined.  We have a total score of 100 based on 36 weighted factors and we are looking at three areas, three core processes where you need to be successful to succeed as a Web site.

• Providing valuable content
• Helping people find stuff and
• Leveraging the feedback loop

Together the 36 factors could be the basis of editorial guidelines or publishing rules.

A lot of large enterprise organizations look at a scorecard as step one in a process that’s about implementing some standards across the organization.

If you have a baseline review you get a scorecard that shows how you are performing today along with a vision of where you want to be tomorrow.  To get there you need to document the rules, so all of your content publishers know what is expected.

We have a lot of experience here and would be delighted to talk to you about helping you create publishing rules as an extra project after the scorecard is completed.

Pros and Cons

An expert review has some pros and some cons.

On the plus side, this is a method that performs really well as a change agent.  You have an opportunity to use it to pave the way to be a standards organization. Grades are a very familiar way to score things for executives, and you get a report that helps you track things over time.

On the downside it is very important that you have a uniform understanding of what the rules are, what the measured points are.  You also have to understand that this is a plus or minus score. Do you do it, or do you not do it?  There are not shades of gray in the way we do the scorecard.

Most importantly you are not involving users in this, it is an expert taking a look at your site.  And you are not really looking at task performance.

Human Performance Testing

That brings us to the next method, which is top task tune up, human performance testing.

To set the stage here, I’d like to call out Professor Barry Schwartz the author of the book Paradox of Choice, Why More is Less.   He makes the point that a lot of times people are judging their own satisfaction based on peak moments, the highs and the lows, and also the end result.

The bottom line is did they complete the task and what stood out — good and bad — along the way?

When we do a top task tune up a standard project looks something like this:

1. First you are going to be recruiting you participants, your test subjects.  No less than eight is highly recommended.
2. Then you are creating scenarios by identifying the top tasks that these folks try to do.
3. Then you conduct the test. We use the Usability Testing Environment, and we do that remotely using the go to meeting collaboration tool.
4. We are going to analyze the results for you and prepare a test report which will be very illuminating.

A quick word about our favorite tool, the applications we use to administer the test, which is called UTE.

What it does for you during a usability test is it allows you to take the test remote.  It expands the geography over which you can take the test.  It also speeds up the collection of data, and it automates the collection of data that might be in dispute.  This eliminates the guesswork of when people complete the task how many clicks it took and the like.

When you automate these sorts of things you can concentrate on analyzing and observing behavior to discover patterns that will improve your Web site.

Select the Right Tasks

Selecting the right tasks to include in your test is a critical success factor.

We recommend no more than seven. We chose the number seven because, frankly your testers are going to lose focus and get a little restless if they have to submit to a test longer than an hour long. Seven is usually the right number to assure you don’t burden your testers.

In securing your participants the thing to think about is whether they are representative of your user community.  You should know going in that there is good research out there saying you can get good results with this sort of test with as few as eight subjects.

The first time you conduct a top task tune up you will be doing it to establish a baseline of performance.  Here you are comparing the results the success ratio of your users against the objectives your organization has established about where you intends the success ratio to be

Ideally after the baseline has been set you go back and retest the same scenarios.

This way your comparison point is against the baseline. When you compare a baseline test with an iterative test you have a much more realistic grasp of the strength of the improvements made on your site based on lessons learned when doing usability testing.

Some web managers are really interested in how they stack up against the competition.  Here what you’ll plan is a top task tune up which compares how your target audience performs on your Web site versus how they perform on a competitor’s site using the same questions.

About Test Participants

Let’s switch gears a minute and talk a little bit about test participants.

You may recall that I said no less than eight participants are necessary when you do recruitment for testers.  I know of some complex site owners who decide they needed 16 participants. They did this because they had a multi-layered audience where they had a real need and desire to get into the weeds and do comparisons on how different audience members would react to the site.  Here, you would aim to split the test group between men and women, and capture info about their age and education level to match the target audience that the Web manager has in mind.

Think about what your own target audience looks like.  How would you cut the apple between gender, age and education targets?

If you do a top task tune up you’ll get a really beefy report at the end of the project which shows the success ratio of your users in performing top tasks during the test.

In addition to the success ratio we are measuring other factors such as number of clicks and time to task.

Your Test Report

In presenting the data, you might show an iterative test where you can see there was some marginal improvement between a test done in March and the test done in November, for example.  Improvements may not be as large as you would have liked, but they will be measurable results.

A test report can also be presented as a nice one-pager that you can imagine being very easy to deliver to executives. It would shows a real measurable improvement between the baseline test and a final prototype test.  I’ve seen such reports presented at a government conference, and I think its a wonderful way to showcase actionable results and show your success to your leadership.  You can include data about the success ratio, how much time it took to perform the task, the number of clicks and how they got there.

And most importantly: capture notes and observations as you listen in as the tester thinks out loud as they are trying to do the task as quickly as possible so your internal documents from the test provide as much information as possible to decision-makers.

Pros and Cons

Any usability test choice has pluses and minuses.

For the top task tune-up, on the plus side, the software is sweet. It is going to measure everything with precision.  Of course, on the downside it has to be setup correctly to collect the data.

On the plus side, the think aloud methodology is going to reveal important cues – lessons learned from the user experience.  On the downside you are missing the visual cues you’d get if you were in the room.

On the plus side, you can do this with as few as eight testers.  On the downside, you have to be really careful about selection bias when selecting those participants and not drop below eight people.

On the plus side, you are going to zero in on the top tasks the things that are really important to your users and matter most.  On the downside, you are going to miss the deeper review of content and navigation that another usability test procedure would give you.

Finally a top task tune up can be repeated over time, as we’ve seen with some of the examples I’ve shown you.

On the downside if you elect to do this only once it is going to limit the impact of choosing this method in the first place

Web Blueprint: Cardsort, Feedback and Benchmarking

The final usability test method in this presentation is what I call the Web blueprint package.

I draw that name from my Web governance model that says there are three phases of Web management- draw the blueprint, build the Web site and manage the lifecycle.    This strategic map gives you a foundation to make marketing choices to reach the right audience with the right content.

There are different times across the lifecycle of a Web site when the time may be right for doing a Web blue print. Perhaps you are at the start of a re-design and need help with navigation labels and structure. Or maybe you want to avoid an expensive redesign altogether and make targeted fixes and adjustments to poorly performing navigation. In other situations, you may need to ramp your positioning in the marketplace and get a handle on how you stack up against competitor sites from a users’ point of view.

If you go down this road, you can expect several deliverables including a card sort exercise, a facilitated feedback session and some valuable benchmarking that compares your site to your top competition.

Card-sorting 101

What is great about a card sort is you are putting your whole navigation through a rigorous exam from your users’ perspective. Does it fit the mental model of how they want the site to be organized?

You want your site navigation to be highly intuitive, simple and easy to learn, and your users can tell you how to do that.

A card sorting exercise is low cost and low tech. You are essentially putting your labels for your navigation — first tier, second tier, sometimes even third tier — all on cards in a nice big deck, and you are asking your users to sort it out and put it in a hierarchy that makes sense to them.

You are going to maximize the probability that users will find what they are looking for when you follow their mental model.

As you compare the results of different testers with the same deck of cards patterns will emerge that will improve navigation.

Remember, it is only one item per card, and you can do this as an open sort or a closed sort.  Meaning you can make some preliminary limits of the categories names you want to sort into or you can let your users decide what category names make sense to them.

Other options are to ask your participants to clarify what the words mean to them to unmask jargon.  You can also ask them to make notations about whether they understood the name and mark it on their cards.  All very revealing data.

A lot of people wonder about navigation. Should I organize my stuff by business group, by information type, by category, by subject, by process?  Let your users answer these questions by doing a card sort. It will also help you collapse parent child categories and create a simpler, cleaner and streamlined navigation. Again, intuitive to users.

Pros and Cons

When you elect to do a card sort as your usability test of choice there are pros and there are cons

On the advantage-side it is easy, it is inexpensive, it is very user-centric. And you are going to get top results to really improve the navigation structure of your site for terminology, labeling and categories.

On the downside, remember you are not doing task analysis, you are not doing best practice expert review, so you don’t get the advantages of those types of tests.

It can be a little time consuming, a little tricky to analyze and you are limited in the number of people who are your participants your testers.

But on balance card sorting can be a really good choice to get you the information you need to improve your site.

Facilitated Feedback

A facilitated feedback session is the last leg of our blueprint package.  And this is something you might elect to do yourself.  But oftentimes we notice large organizations find it very valuable to bring on help for this kind of exercise.

Let’s face it. You have enough to do managing the day to day activities of your Web site. Strategic planning is something that can be outsourced so you have an informed way to make judgments with an independent look about your organizations strategic goals in hand.

Interview key stakeholders you have identified with questions to get issues out on the table.  We like the structure of arranging a workshop or stakeholder feedback session where as a group people can get together and talk to each other about the goals, long-term objectives and what your organization wants to achieve through your Web site. What you get is a valuable, strategic report that illuminates your competitive advantage and where you stand in the marketplace.

Summary

This post has reviewed the pros and cons of a range of usability techniques.  We’ve talked about expert reviews, top task tune ups, the blueprint package, which included card sorting and facilitated feedback.

All of these can be valuable tools in your arsenal as you make the moves to make your Web site a more friendly, more satisfying and more productive for your users.

So if you are ready to move beyond exploring your options and make a commitment to usability testing, we hope you’ll give us a call. Any of the services we’ve talked about today — from expert reviews, top task tune ups to a blueprint package — can be offered as single shot projects, as iterative projects or as a combination of projects.

We think Web managers who commit to Web site usability make a smart choice. You are going to benefit the user experience, make them more satisfied, more productive and put your organization on the road to achieving its goals.

The slides we shared with the audience provide a good overview and case study of how to run an enterprise project with a team to overhaul a Web site.

My Role

With a grant from Library Services and Technology Act (LSTA) to support the work, CCLS fielded an RFP in search of a partner to lead the usability and project management and planning effort for the county library. The goal was to create a virtual library branch and use new technology such as RSS feeds and e-Commerce tools. I successfully competed for the project and joined a strong in-house team as a consultant.

The end result was geared for easy maintenance. Key deliverables included a requirements and planning function, the creation of the information architecture and usability testing. We also set and published the organization’s first Web standards document: Web publishing guidelines to inform content management in the future. We combined listening to both the voice of the customer and the voice of the stakeholder, with a good measure of baseline and beta usability testing.

From my point of view, the work involved in this engagement represents a good overview of what a usability project should look like. The usability expert helps drive the project forward, working in collaboration with the client and the technology vendor. To summarize, the scope of my work included:

Usability

• Expert review scorecard, baseline and beta
• Requirements development
• Benchmarking the competition

Voice of the Customer

• User interviews
• Customer Profiles, including an analysis of analytics
• Task analysis, baseline and beta

Voice of the Stakeholder

• Planning assessment
• Design strategy
• Information Architecture
• Training and Publication Guide

Results

Analytics show visits have grown by 17 percent in the first year. My Web scorecard before and after shows that best practice adoption grew by 24 percent. My usability testing shows that the task completion ratio for top tasks increased by 15 percent.

We transformed problems into solutions:

• The problem of duplication was transformed into a unified look and feel for the content
• The problem of inconsistent page layout, look and feel was transformed by embracing standards to create a consistent look throughout the site
• The problem of a ineffective use of the site to promote the library’s events and resources was transformed by leveraging Web 2.0 to turn make the organization more networked with its members

As a result there are many benefits to the user: the site is more relevant and driven by user’s top tasks, standards help the organization save time and be more effective, and content is more accurate, error-free and timely.

We found a number of strategy elements critical to our success:

• Web scorecard graded the site against usability best practices
• Design Strategy kept everyone on the same page
• Planning Assessment Grid was a strategic road-map
• Web Governance Steps gave the strategy a framework
• Publication Guidelines a desktop reference for content standards

In addition, a number of technology tools helped keep the project on track and running smoothly:

• Basecamp helped with online project space
• Usability Testing Environment streamlined task analysis
• GoToMeeting enabled remote online meetings
• Firefox Add-ons browser tools to help you audit and evaluate sites(free)
• Powermapper automatic site map creator
• Visio Microsoft lets you build wireframes
• Web 2.0 Tools – Google Maps, Facebook, Blogger (free)

Signs of Success

The mark of a successful  project can sometimes be found in the awards and recognition that follow.  Such was the case with the CCLS venture.  In December 2009, the State Library of Pennsylvania contacted the county library leader to inform them that they want to use this Web site project as one of its LSTA Exemplary Projects.  I was grateful too when my task manager, Jonelle Darr, the Executive Director of the CCLS praised my contribution to the team.

“Kathy assisted the Cumberland County Library System with a total overhaul of its web site in 2008-2009. Of all the consultants that we’ve ever hired, she was — by far — one of the very best,” said Jonelle.

“She delivered what was promised, on time and as expected. She not only designed the site’s architecture, but also provided us with the tools and working knowledge we needed to continue to improve our web site. Plus, she was a strong advocate for us when working with our CMS vendor. I would engage her again without hesitation.”

Ignore Poor SEO at Your Peril

Your ability to serve your audience and compete against the noise of other sites vying for those eyeballs is hurt by poor SEO. From a marketing point of view, it is a rookie mistake that needs urgent attention.

I’ve put together a Search Engine Optimization Web site scorecard exclusively designed to provide insights on how your site is performing against best practices. I’ve identified the top meta-data, content, domain administration and link issues that every Web manager should keep their eye on.

Monitoring and fixing these SEO must-haves can make you search-engine friendly. Ignoring a problem on this list could mean your site stays lost in the haystack of the World Wide Web.

How to be Search Engine Friendly

Here is my “sweet 16” site-specific Search Engine Optimization tips to make your site content easier to find:

1. Valid title tag – with keywords anywhere and as first words if possible. Every Web page must have a unique and meaningful title meta tag. Aim for 65 characters in length, approximately 8-10 words. Begin with what is unique about the page and then deliver the contextual information, and not the other way around. Where does it fit in the overall classification?
2. Description tag — The description meta tag should be no more than thirty words long, and should be filled with clear keyword rich language. When you leave out a description tag, or send pages to the server with descriptions that duplicate other pages, you hurt your standing in search engines. You may also confuse users at the search result page, since the search engine will grab the first words it sees and serve up gobbly-gook instead of the description you supply.
3. Heading tags – A Web page is not a word document. The code behind the page should put tags around your headlines and sub-headlines so that they are machine readable and classified as headlines. You should aim to use your keywords anywhere in the H1 Headline Tag, and as the first words where possible. Your other headline tags, H2, H3 and so on, should follow the same principle.
4. Keyword Use in the First 50-100 Words in HTML on the Page – pay careful attention to keyword use and the number of repetitions in the HTML text on the page
5. Keyword use in link anchor text – On pages where you use anchor tags, these links should aim to use keywords to improve your SEO
6. Unique and meaningful links – Every link on your page should be unique and meaningful. That means you should not use “click here” or “more” at the end of a post to get users to follow the link. These are the equivalent of “empty calories” telling the search engine nothing about your content’s value, and confusing users who require screen readers
7. Native formats – Do you use native formats (html, xhtml or xml) for the greatest flexibility for your users? If your site is using flash files without a text alternative your content in that part of the page will not be search engine friendly
8. Plain Language URLs – If you are using a content management platform you may have long URLs which end in a string of numbers and characters. Wherever possible you should employ technology or masking to change this to regular plain language. Ideally, the name should have dashes between each word, which are read as spaces, and be in lower case since not all servers can resolve upper and lower cases automatically
9. Speed – Do you accommodate visitors with low connection speeds to the extent feasible, minimizing page download times for visitors and in most cases, keep your HTML pages under 100KB? A heavy page can be toxic to a search engine spider which wants to quickly move through the Web and not be hung up
10. XML Sitemap – You should include a sitemap.xml file in your root directory. This file should be submitted to Google using your Webmaster Tools account. Have trouble creating this? Purchase a Powermapper license and you can easily make a sitemap.xml file on the fly
11. Employ useful hierarchies – Examine the site architecture of the domain to assure intelligent and useful hierarchies are employed. For example, if you have a top level button called “about” you need to have a landing page for “about” along with any subpages that belong to that parent directory
12. Make all pages searchable – A hierarchy directory structure is needed for search engine spiders to do their work; databases are not searchable — have gateway pages where they exist
13. Remove old content from server – Old content should be removed from the server so only the content you wish to field to users is available. It is always amazing to me what kind of junk I discover out on the server from sites that have are not well maintained. Removing it from your page navigation structure and not the server doesn’t do the trick – you must always delete the file from the server as well or it will continue to turn up on search engines
14. Cross-browser testing – Has your site been developed and tested in multiple browsers (IE and Firefox) and versions, operating systems, connection speeds and screen resolutions, based on an analysis of your audience?
15. Permanent redirect – You should set up a permanent redirect (technically called a “301 redirect”) between the site root and the www address. That way when people skip the www part of your address the search will still resolve to your site. Once you do this, you will get full search engine credit for your work on these sites
16. Really Simple Syndication – Use RSS feeds on the domain, and configure an auto-discovery code in the header tag so the URL displays the RSS button. This will help users link to your content and increase the backward links to your site.

Want a Complete SEO Checkup?

My Sweet 16 SEO list leaves some things out. For example, best practices related to using directories and social media to your advantage are not on this mini-list.

For those interested in a complete SEO check-up my complete scorecard, covering nearly 70 SEO factors is now a special Emerald Strategies service offered to Web managers. You’ll a scorecard and an action plan you can implement right away to boost your find-ability online.

Contact us for special introductory pricing.

NOTE: This article is cross-posted and was originally published on October 2009 at http://www.emeraldstrategies.net/buzz/tips/2009/200910-make-your-site-search-engine-friendly.htm

Story-telling is our native language. The comfortable association goes back to our childhood and defines how we were taught right from wrong, success from mistakes and other life lessons. Is it any wonder it works in adulthood?

Why the Knowledge Conversation Fails

And yet, time and time again we fall into the habits of the disciplines we have been taught in formal schoolrooms and training.

• Don’t forget the chart…
• Show the numbers…
• Here is the thinking-man’s argument…

As Stephen Denning, the former Knowledge Management guru at the World Bank and story-telling consultant tells us, there is a difference in the dialogue between “knowledge” conversations and “story-telling.” Knowledge is seen by society as something that is solid, objective, direct and abstract. Story-telling on the other hand, Denning says, is nebulous, subjective, indirect and unscientific.

So story-telling gets under-valued.

Yet, unlike traditional “knowledge” conversations, telling a story seems to work as a way to engage people and bring them around to new thinking and creative ways of doing business.

When someone shows you a chart, what is your first instinct? Pick it apart, if you are like most people. That column of numbers doesn’t add up. I don’t like the colors you used. We are taught in all our life’s schooling to critique, so that’s how we reflexively react to this type of presentation.

It is not a discussion or conversation at all…

“Let me explain…”

“Let me show you a chart.”

Abstract thinking is tiring, and argument is like war with winners and losers, Denning says.

Stories Get Everyone’s Attention

If — on the other hand — you tell a story, you suddenly have everyone’s attention in a positive, not a negative way.

Story-telling is energizing, refreshing, interesting. It is a “dance” that ignites creativity.

Denning likes to call one particular breed of story a “springboard” story because it can let you take the leap toward a new start.

In a springboard story you have a problem, and the listener thinks: that’s my problem! You have a hero, and the listener thinks: that is just like me. The hero resolves the problem, and the listener thinks: I could do that! The story-teller ends the tale with a generalization that may begin, “imagine a world…” and the listener thinks “why don’t we…”

At the U.S. Air Force, their communication bible called the Tongue and Quill (PDF, 383 pages) has a remarkably similar formula to fix what they label “bad connections” for an officer making a presentation. The fix, they write, is to tell a story.

• First, get their ATTENTION – define the problem.
• Then describe the NEED – using third party validation if possible.
• The next step is to address SATISFACTION, showing the value and benefits that are at the end of the rainbow if the problem is fixed.
• At this point, it is time to use VISUALIZATION with story-telling or best practices to demonstrate it can be done.
• The finale is to describe the ACTION you wish to pursue – the solution to the problem you want the leadership to agree to.

For example, the story may go something like this, Tongue and Quill says: Our state has high tooth decay. This is undesirable and current controls don’t work. Other states put florid in the water. Here’s what our program might look like. Let’s get the health department to act now.

Discuss

If you are struggling to get your message and plan across to your organization using the traditional methods of knowledge worker, it may be time to return to your youth and practice the great traditions of story-telling. Do you think story-telling can triumph in your organization?

NOTE: This article is cross-posted and was originally published on September 2009 at http://www.emeraldstrategies.net/buzz/articles/2009/200909-the-power-of-story-telling.htm

But people often make mistakes in deploying a satisfaction surveys for the first time – and the mistakes often repeat themselves for those who manage existing satisfaction survey programs.

Satisfaction Survey Tips

This is my summary of eight tips to consider if your organization is evaluating the way you measure satisfaction with your Web site.

1. Don’t reinvent the wheel on questions. One is it’s been my experience that a lot of folks try to reinvent the wheel when it comes to your question set (called a survey instrument by the statistics crowd). I don’t recommend it, especially when there are so many good tested questions out there. You want to trend data over time and compare it to several cycles of the same survey to gain good insights on how you are doing. Personally, I am a huge fan of the System Usability Score. Not only has it been found by a UPA study as one of the most effective question sets out there for judging customer satisfaction, but it also has the benefit of letting you benchmark performance (using your overall score) against others.
   

   I think one of the reasons this question set probably works well is it asks a series of questions from both a positive and negative point of view. This makes those taking the survey think a bit more about the answer, in my opinion. So you get fewer responses where the middle option has been selected for every question.

   A 2004 paper presented at the Usability Professionals Association includes a great overview of a series of non-proprietary question sets including the SUS questions if you are looking for inspiration.

2. Why did people visit and did they leave happy? There are a bunch of smart people out there who advocate taking a task-based approach to the questions, focusing on finding out the top tasks people want to complete when they click on your site and if they were successful accomplishing it. Avinash Kaushik of Google, for example, says you should ask only three questions in your survey:
   • what was the purpose of your visit to the site today?
   • Were you able to complete the task?
   • If not, why not?
3. Need a more classic satisfaction metric? Go for the golden three questions. Marketing gurus have for ages used three simple questions to gauge satisfaction:
   1. Were you satisfied?
   2. Would you come back?
   3. Would you recommend to others?

   Bear in mind if you have no insights on why they were satisfied or not satisfied it becomes impossible to remedy anything for those who are disappointed.

   Still, this threesome is recognized widely as the “right” questions to judge happiness in the abstract.

4. Make sure you don’t overdo the sample size. The pushy surveys (like pop-ups) tend to be left on the sites for a long time period and give you tons of replies, far more than you need for survey research. Unless you are doing a heck of a lot of segmentation, if your audience is over 500K you only need 384 (I think that’s the number, if not its close) to be plus or minus five on your confidence level. This is the acceptable target range for actionable results in my experience.
   

   When you report results you should always include the universe or population that participated (usually called the N as in N=384) and the plus or minus info somewhere near the date. It’s even better if you can do a rolling report and show the last four reporting periods (using quarters, months, or whatever time frequently you do the survey) so you can spot trends at a glance.

   Not sure about what sample size to use? A sample size calculator on the Web can help you be a pro.

5. Deliver the survey to a random sample. When you pick your delivery method, be sure to put the need for a random sample on the front burner. Some people shy away from delivering survey opportunities via a Web site. A static survey where the user opts in is probably the least desirable from a statistical point of view. But it does have the benefit of signaling to visitors who see it that you care about their opinion.
   

   The desire for randomness sometimes leads folks to lists of known users (via newsletter signups, etc.). Here, you’d pick a number of people randomly, assume a response rate of a certain percent to determine overall number you need so that you get the plus or minus five percent you are seeking at the end, and deliver an invitation via an email. While it sounds rigorous, this method also has its own selection error issues. You may well find that because the email comes out of context and users may not remember your site that well. So your results are less meaningful.

   Some systems can be set up to deliver pop-ups to every x-number visitor, which is a way you can do random invites with pop-ups, which makes it easier to defend. My preference is to deliver the survey to the user during their experience on your site.

   Other tools offer a permission-based exit survey. Here, the site owner controls how frequently visitors get the invite (i.e. every 10th visitor). It lets visitors easily say no and X-out of the invitation easily to get on with your task if they don’t want to participate.

   When you select your survey technology tool, be sure to explore the vital question of how the system selects who will respond: it matters for the results you’ll be gathering.

6. Time zone settings may be a consideration. For global sites, you also have to be careful about the time zone too. I was involved in one survey where it became clear that because the system was set up to deliver the invite at midnight (something we had no control over) so that you got to the number of responses you needed by morning — ALL the responses were from Europe and overseas! Lesson learned: it’s tricky out there in survey-land. Work with your vendor to know the limits of your tool and what work-arounds are possible.
7. Remember to offer “I don’t know” as an option. Answers don’t always fall neatly into the five scale or seven scale set of possible answers. Sometimes users want to tell you “I don’t know” and if you haven’t left that answer open as an option to them, they’ll get frustrated at the limitations you put around your questions. So do everyone a favor and tack on “don’t know” at the end of the answers you provide in your survey.
8. Explore both paid and free survey tool solutions. Not everyone can afford a proprietary survey tool and their sometimes expensive fee structures. So deploy the solution that fits your needs and your budget. There are lots of free survey tools out there. The paid tools often include consulting services and are less self-service oriented. Many of the paid tools have a free pricing tier as their loss-leader to get you in the door to offer their paid services.

Free survey tools

• Google Moderator
• Poll Daddy (free tier)
• 4Q
• Vizu Web Poll Widgets

Paid survey tools:

• Survey Monkey
  
• Survey Gizmo
  
• QuestionPro
  
• iPerceptions (free tier is 4Q)
• Foresee Results
• Customer Centric Index

Discuss

So if you aren’t doing a satisfaction survey on your Web site, what’s stopping you?

NOTE: This article is cross posted and was originally published on August 2009 at http://www.emeraldstrategies.net/buzz/articles/2009/200908-customer-satisfaction-surveys.htm