INTRODUCTION

Usability testing and evaluation provide insight into how users will likely interact with a particular system.   Frequently usability testing and evaluation are used as part of a process whose goal is to modify a system to become more efficient and satisfying to use.  In the case of virtual reality systems and virtual worlds in particular, there seems to have been relatively little published on the use of usability testing and the learning’s from that testing.  In this blog we look at applying low cost usability testing as part of a process to improve the usability of a new virtual world interface (Nortel’s web.alive).  We also discuss a new variation of traditional usability testing designed specifically for testing group interactions.

What is a virtual world interface?

The term virtual world is typically applied to a computer simulated environment populated by user controlled avatars.  A virtual world interface is the portion of the virtual world application which allows users to view and interact with the virtual world – including controlling their avatar.  It is important to distinguish between virtual worlds and virtual world interfaces as they each have their own usability concerns.  In many ways, virtual worlds and virtual world interfaces are analogous to web pages and web browsers.

Why are virtual world interfaces different?

Virtual world interfaces are different than conventional 2D office application interfaces for several reasons including interactions with 2D image that is a projection of 3D data, multiple users interacting in real time, and the distinction between interacting within the virtual world and with the virtual world interface.

TEST METHODOLOGY

As web.alive is intended to be used by individuals with little or no training and by a population likely to include a significant percentage of users with no prior virtual world experience, usability was a particular concern for the development team from the start.

Usability testing was introduced as part of an iterative design and development process.  Usability test sessions were implemented as new interface features or behaviors were considered in the iterative design process.  Two types of tests where employed: (1) individual user tests and (2) group interaction tests.

1.    Each individual user test used a small number of subjects (typically 5) as recommended by Jakob Nielson.   Each subject was individually run through test protocols targeted at exposing problems in the virtual world interface independently.   Subjects with average business software knowledge were screened to ensure they had no prior experience with virtual world interfaces.

2.    Group interaction tests employed a larger number of users (typically 15-20).  Usability test protocols where designed to expose usability issues with features and functions related to users interacting with each other in a virtual world via the virtual world interface.  Each subject ran through the protocol in parallel so that they could interact with each other.  This did introduce some challenges (see below).  Subjects were screened to ensure that they were capable with basic operation of a virtual world interface to avoid concealing group usability issues with issues related to basic individual operation of the interface.

In either case, results where published to the team within days and updated designs proposed and implemented rapidly to allow retesting in ensure the updated design indeed improved interface usability.

Protocol Design Issues

Group interaction tests require users to interact with many other users.  This creates several challenges for protocol design and evaluation of behavior: (1) controlling the environment such that each user is exposed to the same stimulus is not possible as users must interact with each other, (2) large numbers of users running the test simultaneously would require equally large numbers of observers in traditional testing, and (3) resources where not sufficient to bring all users into a test lab for observation.

Our approach was:

1.    Accept some variability in stimulus, but to design the test protocol in order to provide similar stimulus to groups of users (target of 5) within the test.  For example, usability of the system in satisfying a “locate user” objective could be tested by placing test subjects into pairs where one individual is instructed to hide and the other individual instructed to find their opposite number.  In this case, each half of the test subjects would be running a similar test.  To maximize efficiency, tests where designed such that even when all subjects where not performing the same tasks, all tasks where designed to provide useful usability data.

2.    Accept that we cannot observe all users continuously and instead perform “spot observations”, in test surveys and exit interviews.

3.    Collect data from within the system by observing avatar behavior (as a proxy for the controlling user’s behavior) as well as observations of a subset of users in lab.

OBSERVATIONS

Individual user testing following established low cost usability testing / evaluation and group interaction testing were both effective in finding significant usability problems.   We were also able to determine if subsequent design changes did in fact improve the software.  In several cases, multiple design and test iterations were required to resolve problems found.

Individual user test results

Initial individual user testing found 9 significant usability problems for the new user population with the core elements of the interface.  It required several iterations until these problems were solved (as evidenced by the drop in issues found by test 3).  At that point testing progressed to more advanced/obscure features and new sets of problems were uncovered.

Test    Version    Subjects    Problems
1         b53           5                9
2         b56           6              10
3         b62           5                2
4         B65           5                8
…    …    …    …
Table 1: Subset of individual test result summary data

Group interaction test results

Group interaction testing was successful in exposing new types of usability problems in the virtual environment interface.  We estimate about 30% of problems were group interaction related (GIR) and would not have been found via individual testing.

Test    Version    Subjects    Problems     GIR probs
1         b58           16             15                5
2         b60             8               6                1
…    …    …    …    …
Table 2: Subset of group interaction test result summary data

CONCLUSION

Our usability testing process appears to be paying dividends for web.alive at a time when many virtual world interfaces are being criticized for significant usability problems.  Solutions found to usability problems uncovered via both individual and group interaction testing for web.alive tantalize with the possibility of general solutions and future design guidelines for virtual world interfaces.  Future blogs will explore the solutions we have discovered and the possibility of generalizing these solutions.

Learning is inherent in human nature – something we do like sleeping or eating, fundamentally experiential and social. It’s the ability to negotiate new meanings and involves dynamic interplay between experience and current participation to create a new position. It’s a matter of social energy and active engagement. Learning transforms our identities and our ability to participate in the world by changing who we are, our practices and our communities. We’re all members of multiple communities – within families, organisations, schools, across the web, in sports teams, music bands, cities, districts and streets. Communities of practice are an integral part of our daily lives and offer powerful opportunities for converting information to knowledge to useful experience.

I’m fascinated by the question of how organisations support and spread learning and how web.alive will do that on a wider and more effective scale – because make no mistake it will. Already I’ve seen how the global nature of web.alive strengthens the local – earlier this week I met with UK customers in the Lenovo elounge where a shared interest sparked discussions that will lead to local interaction.

The global nature of web.alive doesn’t replace the local but strengthens it.

Our Communities are not static and web.alive provides them with the opportunity to grow, evolve and achieve their aims. Web.alive allows the latent, inherent similarities and shared objectives of its users to connect and collaborate in a way that fosters understanding, teamwork and community.

Organisations and educational institutes are social designs directed at practice – it’s through their practices that they can do what they do, know what they know and learn what they learn. Communities of practice are therefore key to an organisation’s competence, its development and success. We are social beings who look to learning to provide meaning. That meaning has always been shared in communities of practice. The web.alive environments we build will forge new communities of practice that are truly global - where membership is based on a shared desire to co-operate, create and collaborate. Our similarities will inspire interaction and our differences spark innovation as we rub shoulders with those whose personalities and perspectives are different from our own.

Web.alive creates more effective and engaging corporate and personal communities where learning objectives and shared interests will grow and evolve through the experience, not be limited but liberated by its possibilities – surely that’s a good thing - Let me know your thoughts…

Thursday came and although there were a few last minute scrambles to ensure that everything looked just right in the Aquaknox restaurant at the Lenovo Hotel (the team in Ottawa had everything working, it was us at the biggest electronics show of the year who could not find a spare power cord) we knew we were set for an exciting day both in Las Vegas, in the labs in Ottawa and on the road in London.

People came by steadily throughout the day on Thursday and at about 3PM our news releases hit the wire and we finally announced to the world that we were proud to partner with Epic and had licensed the Unreal Game Engine and were even more excited to announce that we had launched the first public web.alive site for our customer in Lenovo.

The news spread quickly throughout the blogosphere and things seemed to pick up at the show as well. The next few days were extremely kind to us with a tremendous amount of positive coverage from a wide variety of sources.

We are excited for you to check out the Lenovo E-Lounge (www.lenovo.com/elounge) and hope to meet you there.

web.alive In The News

Nortel Bags First web.alive Customer

Nortel adds Epic gaming engine to virtual world

Nortel: web.alive Platform Is Live

Lenovo buys into Nortel virtual shopping app

Nortel’s 3D Virtual World Technology Powers New Lenovo Shopping Lounge

Lenovo Using Nortel’s Web.alive for E-Commerce

Montreal firm displays 3-D video innovation at global tech show in Las Vegas

Lenovo Rolls Outs Nortel’s web.alive Virtual World Application

Nortel uses Unreal Engine and Lands Lenovo as a customer of web.alive

Nortel and Epic Games Improve 3D Environments for the Next-Gen Workforce

Nortel Rises Again on Virtual Store News

Nortel Licenses Unreal Engine for Web.alive; Opening Up To Development Community

Nortel’s Web.Alive Debut with Lenovo at CES

Review: Lenovo’s eLounge Virtual World

Overheard - Move over Second Life. Web Alive has come to town

Web-alive…

A new Unreal virtual world dawns – Nortel uses Epic Games engine for Lenovo’s ‘eLounge’

Gears of War Engine Enters Business World Through Nortel

It’s About Engagement

“Organizations are saving vast sums of money by bringing the work to the people rather than the people to the work”

We know this to be true of every large organization that has a dispersed workforce and is conscious of costs when it comes to real estate, energy and transportation. Workers either work from home on a full time basis or they may even work in a remote satellite office, either way they are not physically connected to the people and place that they call their work home.  This space and time zone difference can create a significant disconnect with their peers and affects job engagement. As noted by Driver, “People who work away from the center of influence (e.g. headquarters, facility where the boss works, etc.) are inevitably second-class citizens who have less access to people and tools than their centrally located counterparts…How many times have you dialed into a conference call and been forgotten by the people who were sitting around the conference room table?“

Driver goes on to report on an October 2007 study which found that just 21% of the global workforce surveyed are engaged in their work - meaning they are willing to go the extra mile to help their companies succeed. Nearly 4 out of 10 (38%) are partly to fully disengaged.

Those numbers are simply staggering and therefore it would seem that all businesses have cause to find ways to better engage their dispersed workforce.

As concluded by the ThinkBalm report, “Virtual worlds and campuses, immersive learning simulations, serious games, and 3D business applications deeply engross the user and give people experiences that are perceived as real even though they take place virtually.”

We have seen this in our own experience with web.alive. In group interactions when the person speaking moves their avatar in one direction or another (simulating turning from side to side to address the whole audience) you will note that at least 9 of 10 people will track those movements with their own avatar. They are simulating the real world and they are for the most part not even conscious that they are doing so, yet they are engaged in the discussion and it shows because they follow the speaker. The 10th person is the one multi-tasking to something else and is easily shown as not being engaged in the discussion. A very simple measure of the ROI when it comes to engagement, but a very effective one.

See The Invisible

“The Immersive Internet allows us to do things we can’t do in the physical world, and can’t do - at least not as well - using other forms of communication and collaboration technology. The Immersive Internet allows us to:

– Work in 3D - wherever we are
– Interact with others spontaneously - wherever we are
– Work with a big group to create 3D mind maps and workflow charts
– Collaboratively build ‘quick- and-dirty-’ prototypes with remote colleagues”
– See the invisible and do the impossible”

These five items very neatly sum up the possibilities when it comes to the use of virtual worlds. From our own experience internally with web.alive I can tell you that the ability to work in 3D spontaneously has increased collaborative efforts on our team ten fold. Not only can I easily jump into a meeting with a colleague in a virtual building within web.alive but I can leave all our confidential work on the walls and simply lock the building on the way out so that only my colleague and I have access. Later if either of us wants to update the work we just log back in and enter the building which houses our work. This is quite similar to the concept I wrote about in an earlier post.

Taking that a step further to what Driver describes as “see(ing) the invisible and do(ing) the impossible,” technicians are able to visually observer temperatures, air flow, energy flow, heat maps and even walk through virtual data centers transmitting real time, real world data.

The Immersive Internet opens the door to things simply not possible using alternative methods and at the same time Immersive and Engaged go hand-in-hand. It’s hard not to like it (or agree) when Driver concludes “While the adoption of the Immersive Internet in the enterprise is still in the seeding stage as of late 2008, it will become mainstream within five years. By this mean that it will have a significant installed base within Global 1000 companies and large public sector organizations and at least a few sucessful large-scale implementations.”

Over the days / weeks and months since that decision, I have progressively come to appreciate the value equation and have a genuine excitement of the prospects of virtual world communication and specifically web.alive both in terms of what it is and what it will become.  With each new customer interaction new uses of the tool are exposed.  Customers appreciate the diverse communication which can occur in the environment and can leverage this into many different aspects of their business; training, customer support, collaboration, marketing etc.  As a web plugin, the ability to coexist and interact with other web applications in a seamless manner and or the ability to be added to a customer’s existing web site makes the introduction of this technology a natural evolution to the existing web.

The ability to communicate via the web as we naturally do in the real world without being limited by communication specific technology (phone call / conference call / web cast etc.) will fundamentally change communication.  Examples of this include being able to talk to someone while listening to a presentation without requiring an Instant Messenger session or having the ability to share information without requiring some additional application while we talk on the phone. Also, knowing who is speaking or making noise (done visually within the environment) without having to interrupt a whole group greatly raises the efficiency of the communication.  These are but a taste of where this technology will lead but the real question is how this will play out?

It is clear to me now we’re on the cusp of the next communications revolution.   Reaching out to the Gen-Yers is the easy part - this is a variant of the technology they interact with daily.  The question is will the Gen-Xers and Baby Boomers “get” it.  From my personal experience and the interactions with many customers to this point the answer is absolutely yes.  Only time will prove this out but in the meantime here is one Gen-Xer immersed and excited about virtual world communication

Some highlights of the story

• The application is ’software-agnostic’, so it will run in most corporate computing environments, and it doesn’t require any particular Nortel hardware. As Hyndman explained, “It will run on almost anything you’re likely to have in the enterprise or for that matter, most consumer equipment. That is actually one of the challenges that we’ve overcome, doing 3D graphics on these older machines that don’t have dedicated graphics cards. It’s a challenge and it is one that Second Life struggles with for example.”

• With the responsibility for bringing web.alive to market as a commercial product, Hyndman has already incorporated the software into his own working world. “I use it every day. I have it up and running right now and I was just talking with some guys about the project,” he said. He uses web.alive instead of the company telephone and e-mail systems to communicate with his own team. It is not yet widely available across Nortel but the company is starting to roll it out to other employees.

• Like other virtual worlds, people in web.alive take the form of cartoon-like ‘avatars’ when they enter, but Nortel believes that in most installations, they will maintain their own identities, rather than assume new ones. The goal is to place the user in a real social or business context, not a make-believe one. As Mr. Hyndman explained, web.alive is not designed to be an escape from this world but rather another way of living and working in it.

Monday morning, Central London - It could have been something Arn and the guys had put together. Standing on the steps outside the conference centre I was looking through the rain at Westminster Abbey in front of me, Big Ben and the Houses of Parliament (of HP Sauce fame) on my left and the overarching Millenium Eye (in i-pod white) rotating around the proceedings.

But it wasn’t something the web.alive team had developed – if I walked into Westminster Abbey I couldn’t be greeted by an avatar that would explain the history and intrigue of the building to me, I couldn’t climb Big Ben or enter parliament to ask Gordon Brown what was going on with the banks.

So as I pressed my metaphorical F4and went back inside my own skin I turned and entered the conference with the thought that the real world was good, but was it always good enough ?….. I’d carry that thought for the 2 days of the conference…..

We’d been invited to participate in a panel discussion on the future of virtual collaboration in the enterprise by the organisers – I’d spoken with them the night before and had been briefed by Arn around the view we’d express – I was raring to go.

The Conference kicked off with a key note by Mark Kingdon, CEO of Linden Labs – I was listening to the words but on the lookout for the message, as interested in where their compass arrow was pointing as to the distance they’d travelled. Before long the message was taking shape – we’d grown up in a world of flat data and of 2 dimensional technology experiences. That had changed and Linden Labs saw themselves slap bang in the centre of this new world – ok, good, I’ve got it – but where was all this going, I understood his points but what were we going to do with this new world, which way would it spin, what would its continents look like, how deep were its seas, how bright its skies and where would the motivations to inhabit and explore it lie.

Before long the gravitational pull of Virtual Worlds started to become clear – Mark Kingdon spoke of potential learning application, Steve Prentice of Gartner reminded us that it was all about people and the tone for the conference was set - the inspiration and application of virtual worlds would lie in the human experience.

Good, this set things up nicely for our panel discussion on the future of virtual collaboration – I took my seat next to Bernard Horan (Ber-nard for our US friends) of Sun Labs, Chris Collins of Linden and Rupert Key from Malden Labs who had just got back from his holidays that morning. Before I’d sat down they’d each asked if they could talk with my about web.alive during the morning – a good sign. We politely took our turns to answer the moderators questions on the future of virtual collaboration, its obstacles and opportunities. I diligently inserted the key web.alive points (honestly Arn) and waited for the audience questions. It’s the sign of a good event if the audience questions are more interesting than the panellists answers – and this was the case here, behind the questions was a common enquiry, how can we build the bridge between yesterday’s user experience and the one we can offer today.

I’m convinced that our team can be those bridge builders, we can connect one communication era with the next – in fact, we’ve already done it.

Day 1 continued apace – even when the discussions were headlined by a focus on ROI, Interactive TV, Branding and Advertising the pendulum of discussion always swayed back to a consideration of collaboration and interaction as the key to virtual world adoption

Day 2 dawned with the promise of continuing the discussions and debate on everything Virtual world – it didn’t disappoint. Niniane Wang of Google kicked the morning off with a keynote that articulated the impressive nature of their presence and desired reach for all things web. As I listened to Niniane I was struck by a point – were we talking about virtual worlds at all or were we hearing a manifesto for the creation of the 3D web. I started thinking about the history of the web we’ve seen and incorporate into our thinking and creations. I thought back to 2 computers being connected – packets being sent from one to another – then to pages being linked together, then data to data connection and wondered how we can best link ideas to ideas. With the proliferation and power of mobile devices and improved connectivity our architecture and platform decisions will be important.

I was brought back down to earth with a 2nd afternoon agenda peppered with legal and policy considerations and conversations – just what I needed to get my feet firmly back on the ground.

And then before we knew it the event was coming to a close. Introductions had been made, discussions held and connections forged – more importantly we’d had the opportunity to discuss web.alive and to get a feeling for where the market is going or needs to be led. Direction and distance have not been definitively decided and web.alive has got an important voice. Throughout the 2 days of the conference I heard a constant tone and conversation – Virtual World are here to stay and will become mainstream. How and what speed that will happen is up for grabs. It’s going to be about collaboration, community and experience – web.alive is in a good position …..

Tuesday evening – waiting for the train at Victoria station I wish I could just teleport home from here … Arn and guys – can you put that on the plan of record…..?

A while back, Christian Renaud posted an interesting question “Riddle me this…” on his blog:

Fast forward to the office of the future, the virtual office version.  You have packetized spatial audio.  You have user created content.  You have streaming video and powerpoints and presence information.  You have ever-changing mixes of synchronous and asynchronous traffic types all over walls and tables of your virtual headquarters.  This is much more bandwidth intensive than Warcraft, and if you are having a staff or funding meeting, the voice/video latency is arguably more critical than simple ‘the dragon killed you before you hit it with your sword’ telemetry data […]

So, given that you can’t ’shard off’ the different offices (you are trying to facilitate the ‘death of distance’, right?) and you need to have a single virtual office where people can intermingle and interact richly, how are you going to accomodate the insane bandwidth and latency requirements of a fully-annotated virtual environment? Or, are the geographically remote participants going to have to suffer the latency inherent in such a rich environment as the cost of global collaboration?

This is going to be the multi-million dollar question as virtual worlds continue to mature. I think that at least part of the answer is going to be the same as those developed for existing web media delivery. That is, cache the media close to the consumer. This solution works well for web pages, graphics and streaming video; the same solution will certainly work just as well for delivering the relatively static 3D content and media that makes up the bulk of the data required to render a virtual world.

This leaves the problem of dynamic user created content (3D content that must be visible to other users as it changes in near real time), application sharing, avatar telemetry, text messaging and voice. Of these, I would suggest that dynamic user created content, avatar telemetry, and text messaging are relatively low bandwidth and thus easier to solve. The challenge will be with application sharing and voice.

For both application sharing and voice, many current solutions use peer-to-peer communications. This architecture has several benefits: (1) no network or cpu cost for the virtual world host (2) good performance for small numbers of peers that are near each other in network topology sense. Unfortunately, peer-to-peer solutions don’t scale well to large numbers of users. In the worst case, n users communicating to each other results in n x n streams – naturally, the bursty nature of voice and application sharing allows optimizations that generally avoid this worst case scenario, but regardless, simple peer-to-peer communications present significant scaling problems which I believe will lead to their eventual elimination from use in enterprise virtual worlds.

It will be interesting to see how solutions evolve over the next few years to handle the problem of scaling voice and application sharing applications for global enterprise virtual worlds.

Two of the facts that jumped out at me and my thoughts on how I see them.

- 61% of CEO’s admit they do not know enough about emerging technologies.

I think that while this comes across as a high number, the fact of the matter is that while CEO’s do need to know about emerging technologies and understand what is going on at the ground level, the best ones have people around them that will always know more about emerging technologies than they do. Read any good business book or talk to any respected CEO and they will all tell you that the most important aspect of their business is having people compliment their areas of strength and provide expertise in their areas of weakness.

- 56% of senior business people have never logged on to Facebook

For me this is more an issue of fear of technology and misinformation that some of the new services are just for kids, teens, young adults etc…I believe that companies need to push the adoption of these new technologies in the workforce not because they believe that they will result in a certain ROI, but more because this is how they will attract and retain new talent while at the same time engaging their existing more senior management with the new hires.

I think this point deserves a bit of a deeper dive as it relates to virtual worlds and web.alive specifically. As is mentioned in the article, many companies have used virtual world technology for everything from team meetings to recruiting new hires. The value of these types of events has certainly be shown by others, but I wanted to comment on the fact there truly is a level of connection made between high level executives and the rest of their organization when interactions take place in the virtual world. In web.alive while everyone knows immediately who they are talking to (because of our user authentication), the apprehension you may feel at approaching even the CEO to ask a question or even just engage in casual conversation is absolutely gone (at least for me). Although I still would not approach a CEO for no reason, if and when the opportunity arises I certainly do not hesitate to walk over and strike up a meaningful dialogue.

“In coming years you’re going to look at what we have today and think it’s kind of a joke.” - Bill Gates

Bo Gowan of Nortel buzzboard has also done a fabulous interview with Keith Weiner of Diamondware which I point you to.

So for now please grab a pair of stereo headphones and listen to the future of audio.

Click here for a link to the 3D Audio Demonstration