I was honored to be invited to the conference to conduct a spin-off of my UX Show and Tell workshops. The audience participation proved once again what I’ve been seeing across the country and into Toronto and London: that there really is a pent-up demand for user experience designers, interaction designers, information architects, and others to pull back the curtain from their work and share the goods or ask for help.

But as my session didn’t occur until Sunday, I had plenty of time to catch up with old friends from DC, my recent colleagues in Philadelphia and New York, and to finally meet a number of people with whom I had either communicated but never met or simply didn’t know until our time together.

Though I took away a lot of important insight in the sessions and keynotes, my real appreciation of the time lies in the moments spent forging new relationships or re-galvanizing existing ones.

My personal highlight of the conference occurred Saturday morning, when my friend Jeff Parks asked if I could sit in on a low-key discussion about design research with a few other folks. It wasn’t until a few moments before the UX Workshop’s video cameras went live did I realize I’d be having such a discussion sitting next to Indi Young, Eric Reiss, Daniel Szuc, Steve Baty, and of course Jeff himself.

Our conversation (which will be posted to www.theuxworkshop.tv thanks to sponsorship by New Hampshire-based Mad*Pow) spanned numerous topics involving design research, such as knowing what to investigate and how to dig deep enough, convincing clients when you need more research or perhaps even less, and much more.

Not only was the discussion itself valuable, it also granted me the opportunity to finally meet some of that Mad*Pow team, including Amy Cueva and Megan Grocki. I had been aware of their stellar work for some time but had never crossed paths with any of their team in person.

My workshop was a success despite a curveball at the last second: though we were planning on a discussion-style format, we couldn’t get the room converted from a presentation-model layout to round-table in time, and because lunch required the round-tables we’d have about 45 minutes for Show and Tell (usually the workshops are about 2 hours or more).

Will Evans quickly volunteered to kick off the Show and Tell by discussing some of the deliverables originally appearing in his “Right Way to Wireframe” workshop held on Thursday of that week. Will’s work process quickly captivated the audience and also showed the crowd that discussing your work doesn’t have to be all that painful.

Other presentations followed, ranging from people looking for help with their design approach, to other folks who wanted to walk through prototypes of their work to get feedback.

I’ll be the first to admit there were some unique challenges to the Show and Tell, but fortunately with a bit of adjustments I’m convinced such a workshop has an important role in an international conference attended by practitioners of varying levels of experience and expertise.

Looking ahead, I want to continue to build the UX Show and Tell brand with more of the connections I established in Savannah, and I’m sure the natural partnership with the IxDA will provide such a fertile foundation for growth.

Professionally, I can’t wait to engage with this crew in the immediate future and beyond, in any capacity, be it on project work, over a few drinks socially, or again on the conference circuit.

From a professional perspective, I’m beginning a long-term consulting position as a UX Strategist with a major pharmaceutical company to help design a socially-rich enterprise intranet application.

The personal perspective is equally beneficial: I’m really excited to move my baby girl closer to my wife’s parents (my Mom lives in Indiana and my Dad is in Hilton Head, South Carolina) . In addition to my in-laws are numerous cousins, aunts, uncles, and my wife’s two brothers who I’m closer to than some of my friends. If it takes a village to raise a child, then I’ll trust this village of family moreso than the alternative.

So between the great work environment, both for this current gig and whatever comes next, not one but three rich UX communities, and the opportunity for my own family to be closer to more family, the decision really wasn’t that tough.

I’m also looking forward to expanding my network of user experience professionals to include New York City, Philadelphia, and New Jersey, and rest assured I’ll continue rebel rousing the UX community to share their work at UX Show and Tell workshops throughout the region.

But that doesn’t mean UX Show and Tell is leaving the Washington DC area. My friend and fellow UX designer John Douglass will pick up the flag and facilitate workshops in the area. He’s already accepting reservations for our next event on January 12, 2010.

I’ll certainly be back in DC soon, but I also can’t wait to see more of you in Philadelphia and New York for UX Show and Tell, and I encourage you to join me for for lunch on Sunday at Interaction 10 in Savannah, Georgia.

This workshop was unique from its predecessors in format, attendance, and dialogue. Specifically, as Show and Tell has previously consisted of about 10 people around one table, the Baltimore Parlay event attracted around 20 people in an open-air studio setting, with presenters showing their work at the front of the room.  And due to time constraints, two people shared work as the accompanying dialogue expanded from exclusively discussing the work to also why the work was valuable, how it could be sold to skeptical or budget and time sensitive clients.

Ren Pope preceded the Show and Tell with his working draft of his proposal for the IA Summit in April.  He shared strategies and processes for understanding information architecture’s role in an enterprise environment, and in the nature of Show and Tell, asked for and received feedback of the presentation’s flow, content, and his delivery.

Following Ren’s presentation, Jim Jarret presented a K-J diagram, an affinity diagram useful for identifying important data among ancillary information often discovered during brainstorming exercises. Jim discussed when to deploy the diagram and a step by step summary of the process.  Attendees’ questions ranged from how to get senior management on board to support the effort to whether the diagram itself could be understood by business analysts, product managers or other audiences.

Elizabeth Randolph of Better Experience shared paper wireframes and prototypes she tested earlier in the day. While some folks were familiar with using low fidelity paper wireframes to generate early feedback, others were curious how to replicate online interactions that may not map well to paper, such as toggling the visibility of panels on the screen or excessive scrolling.  Others were curious to hear how printing and testing computer-based wireframes compared to using the same wireframes in a clickable prototype on screen, whereas others were also curious how to get client sign-off on low fidelity wireframes when they’ve been conditioned to accept high fidelity mockups.

The responses following Show and Tell have been tremendously positive, ranging from people asking when the next event will be to telling me they’re already identifying what work they should bring next time.

The next two UX Show and Tell workshops are November 17 at Viget Labs in Falls Church, Virginia, and at P’unk Ave in Philadelphia on November 18.

If your organization would like to host a UX Show and Tell, contact Chris Avore at (202) 281-4573 or email him at avore@erova.com to learn more.

UX Show and Tell is an informal workshop that’s all about the work, where designers can share feedback and ideas on strategies, outcomes, and deliverables, and meet other practitioners in a focused but relaxed environment.

Show and Tell participants brought a number of unique deliverables, such as concept maps and task models of complex web sites,  a process chart detailing how, where and when to integrate UX in an Agile software development lifecycle, and interface design mockups of a scheduling application.

I started the workshop sharing a deck of documents I used to provide visual conclusions from a number of user interviews and observations I conducted for the Library of Congress.  As user research continues to grow in recognition and importance, I thought there could be some value to *showing* how users behave rather than simply describing behavior in a Word or Powerpoint report.

…participants should be able to share specific solutions if they’re aware of the problems in advance…

I was hoping that the group would bear with me since it was the first workshop.  And not surprisingly, there were bumps in the road, from logistical issues such as starting on time to ordering food, to procedural hiccups such as how and when questions should be asked so as not to derail or sidetrack a presentation.

Fortunately, I’ve already begun incorporating participant feedback into future Show and Tell workshops.

For instance,  I’ll ask participants to identify what they’d like to share or problems they need to address during the RSVP process so other participants may be able to help. While I don’t want to see the workshop be so targeted that one session will only be dedicated to wireframes or another exclusively committed to persona documentation, participants should be able to share specific solutions if they’re aware of the problems in advance.

By identifying an issue first, the group can provide more direct, concise feedback.

Moving forward, the workshop will also run more smoothly if each participant quickly says what they’re sharing and what issue they’re having or where they’re looking for feedback.  During the first session, a few conversations began to wander off course when feedback that was meant to be helpful wasn’t entirely appropriate to the project or the project’s audience. By identifying an issue first, the group can provide more direct, concise feedback.

I also see the benefit to posting what was presented on a wall or whiteboard so the group can take a closer look at the documents. The photograph above shows how a number of deliverables were spread out on a conference table which certainly worked, but won’t scale well or support electronic presentations without a projector.

If your organization would like to host a UX Show and Tell, it’s really easy.

At the suggestion of one of the participants, I created a Google spreadsheet to list who attended with contact information to help expand each other’s network of practitioners in the area, many of whom had never met before.

I’m already discussing new locations to host workshops, both in the immediate DC area and beyond. If your organization would like to host a UX Show and Tell, it’s really easy.  Just have space for about 8 to 10 people, a table and chairs, and now a whiteboard, and we’ll set up a date.

Joel Spolsky of Fog Creek Software recently claimed on his blog that “when people are successful at controlling their environment they become happier, and when they can’t control their environment, they get grumpy”.  This can certainly be true. But does this mean we, as experience designers, have to concede control to the user to provide a favorable experience?

Let’s take a look away from the glow of the laptops and monitors and go outside for an appropriate, though perhaps seemlingly reaching, example.  As an avid golfer, I’ve played a number of courses where the layout of the course is prohibitive to walking.  As a result, I’m required to use a golf cart, many of which are equipped with global positioning system software.

The GPS system often features scrolling sports scores, the club house’s menu with one-touch ordering, the ability to summon the on-course beverage cart, and of course the layout of the particular hole we’re playing, complete with multiple views of the fairway, green, and so on (the blog post discussing these GPS golf interfaces is for another day).

While these features are a great amenity to the experience, it also allows the pro shop to monitor exactly where I am on the course and to confirm I’m keeping up with the pace of play.  The software also regulates where I can drive on the course, either to protect its property (by forbidding the cart to drive onto greens or into bunkers) or to protect me (by limiting the speed of the cart, particularly on steep slopes).

In short, though I feel like I can drive the cart anywhere and as fast as I want, the software system is actually well in control of the environment but offers me an implied perception of control.

Such a paradigm of perceived control works well in the software space as well.

When designing software, the ideal experience does not provide total control by the user, but a perception of control by offering relevant, context-aware tasks and content to the user.

For instance, many online shopping cart and registration process flows remove the site-wide navigation to limit distraction and to focus the user’s attention on the imperative task.

From the business’s perspective, regulating what users can and cannot do can protect users from themselves and prioritize important actions or content.

See the screenshot from the familiar Amazon.com checkout experience as an example of limited navigation.  The user cannot begin browsing for another book here and risk abandoning the cart. Likewise, the user cannot undo any information that’s previously been submitted which could confuse and aggravate the user.

In other cases, an adaptive interface may be instrumental in providing users a feeling of control.  After all, if a fledgling investor just opened an account with $1,000 in a money market fund and identified himself as a novice investor (required during brokerage registration), there may not be a strong need to offer foreign currency exchange research as a primary option.  Of course, such links to currency exchanges are available, but not as prominent as more common tasks based on the persona model of a novice investor.

And while much of Spolsky’s article really cites the benefits of system feedback and not user control, there are a few strong examples that outline how we can design software that provides this perceived control.

For example, Spolsky discusses how implementing AJAX allows his software’s users to manipulate table columns and employ keyboard shortcuts.  Indeed, the ability to drag or hide table columns or send keyboard commands to the system can certainly instill a feeling of control in a user.  If these features were designed to be available after the user was familiar with the system then that sense of control should be heightened, as the user will have developed his confidence in his skills and mastery of the system.  Immediately providing these features risk burdening the user with distracting glitz.

So while it’s certainly true users may say that want full control of their software, the systems that offer a perception of control while monitoring how its users interact with it will have a sturdier foundation with which design a better experience.

Paul Sherman’s recent article Usability Testing ≠ A Good User Experience attempts to establish that usability testing doesn’t guarantee good user experience.  Claiming that the only reliable method to developing a good user experience is “strategic design”, the author asserts that focusing on “tactical usability” is a myopic exercise.

I certainly agree that rigid, task-focused usability testing is unlikely to glean new insight into a product or service’s user experience.  And a usability test can often hinge on the quality of the script and system and the personalities of the moderator and participant. But does user testing only elucidate whether people can successfully complete a given task? I have my doubts.

Dr. Sherman’s argument that usability testing doesn’t quantify user enjoyment is not new. Long before User Centered Design principles were accepted in development teams, software interface designers and developers still needed to assess the usability of the systems they unleashed to their clients.

In an effort to de-humanize usability testing and provide quantitative results, engineers often relied on GOMS (Goal, Operators, Method, Selection) and Keystroke Level Models to assess the usability of existing systems.

Often these findings would be presented in numerical reports claiming that Feature A was 1.4% more usable than Feature B under strict, predictable circumstances.  Such tests proved effective enough for years of practical use, though today they almost seem quaint when compared to the user-driven research practices of the present.

Today, most software developers and their management teams need and want usability tests to reveal how people feel about their application, while also confirming their audience can successfully complete step 2 before attempting step 3.

Usability scripts can elicit how people feel about a system, and a moderator tasked with observing participants’ interactions and emotions can also glean valuable insight, at times even tabling task-based questions to prompt a particularly effusive participant to expand on how she feels about the system.  Granted, you can’t “tactically” assess those emotional reactions, but it’s also safe to say it’s difficult to fundamentally quantify any user experience as well.

Dr. Sherman continues that usability testing cannot “craft a unified user experience, plan for tomorrow’s user experience, [and] create delight, loyalty, and stickiness”.  Well, that’s certainly true.  But a usability test or other user research can identify if the user accepts and welcomes the unified experience from screen to screen and through the system’s interactions.

And a usability test can also provide insight into how to build future releases of software or web sites as well, or what may not be important at all.

Perhaps most importantly, I could argue it’s the usability test that will prove whether the designer is creating a system that is capable of instilling delight and loyalty.  I’d much rather stand behind a system that’s been put in front of my target audience and generated feedback than trust the words of my designer that he sprinkled some “strategic design” into his workflow.

Back in 2007,  UX Matters published Connecting Cultures, Changing Organizations: The User Experience Practitioner As Change Agent wherein Dr. Sherman wrote

Above all, our goal is to bring the users’ wants, needs, abilities, and limitations into the organization, and ensure that the organization doesn’t forget the user during any of the design and development stages

In my experience, usability testing is a prime opportunity to confirm your system is built with an understanding of those wants, needs, abilities and limitations.

The magazine is an open collective talking, sharing and finding answers about the interaction between people and products, systems or processes.

McCullough himself said that while new reviews have been limited (which is understandable for a 5 year old text),  few reviews have been “this helpful”.

Read the complete review and other insightful articles by such authors as Jeroen Van Geel, Will Evans, Adrian Chan and Steve Baty at Johnny Holland Magazine.

Over 45 minutes, we discussed an introduction to human factors, cognitive psychology, and interface design heuristics. And since Mr. Brown’s class is in the beginning stages of a web design project for a local business, we also used the time to focus on identifying audiences, task modeling and other discovery-themed processes.

Mr. Brown has done an exemplary job at recruiting phenomenal speakers to talk with his students–to the point that many of his speakers contact him requesting to speak (I fall into this camp).

His students certainly appear to enjoy a dialogue with practitioners who have real-world experience that they can immediately apply to their own schoolwork, and hopefully put to use for years to come.

Hopefully more high schools, community colleges, and universities will begin or continue to interact with the technology community to benefit their students.

But us practitioners cannot wait to be contacted.

Instead, we should introduce ourselves to those who could benefit from our experiences.  Yes, it’s personally rewarding. But more importantly, those experiences can shape the thought and actions of our future interns, colleagues, and partners.

In the context of  user interface design, even the term personality itself can be confusing.  Luke Wroblewski’s Design Philosophy summarizes that

The combination of a site’s organization, interaction, and visual presentation creates and supports the personality of a Web site…

and that

an appropriate and evocative personality not only tells the right story to your audience, but it also provides distinction, appeal, and wholeness to your site.

In short, tailoring navigation, layout, color, and messaging to your audience shapes not only how they’ll use your web site, but also how they’ll feel about the experience as well.

Audience Considerations

However, some people may argue that many web sites or applications are simply utilitarian: the user should be able accomplish his or her task without inconvenience.

Others may claim that spartan software with more functionality will be more appealing than software with fewer features but more personality.

But before we make sweeping generalizations that these people will always like that software and those people will always approve of this software, we must remind ourselves that the personality of web site must be tailored to its users.

Do EKG monitors need personality? Or do heads-up-display interfaces in jet aircraft cockpits require a fun interface? Of course not.  The priority shifts from providing the user that aforementioned “appeal and “wholeness” of the experience to relaying critical information free of distraction or clutter.

In other words, systems need to balance useful functionality with a well-planned strategic personality.  Unfortunately this is easier said than done.

Case Study

I recently purchased a Garmin Forerunner 305 to track my training as I prepare for four triathlons, a few ten mile races, and a half-marathon this year.  It’s a GPS-enabled personal training device that can monitor my heart rate, distance, time,  an estimate of calories burned, and a few other assorted data from each running or cycling workout.

While the device’s screen provides summaries of each workout, the device sends the GPS information and each workout’s data to my computer where I can view the relationships of each workout or a number of workouts over time.

Garmin provides its own software for analyzing my workouts, but the data can be imported by a number of other applications that aren’t developed by Garmin or its partners.

We’ll examine Garmin’s own Garmin Training Center, Berbie Software’s Trail Runner, and SportTracks by Five Zone Software, plus the less-robust Nike + system to determine if these user interfaces’ personalities enhance the user experience.

Welcome/Home Screens

The Garmin Training Center and Nike + display a home screen, whereas Trail Runner and SportTracks show the latest workout.

Garmin Training Center

Unfortunately, the Garmin Training Center default screen is devoid of any personality or utility. It simply reminds me I was born 31 years ago and my weight from when I entered it in January, and my name.   In addition, there are also numerous codes and zones, though they’re not tailored to my performance.

Nike +

The Nike + home screen takes a different approach.  Instead, the welcome screen displays the latest workout, complete with a Flash-based animated jogger showing bouts of extended speed and exertion.

Here we can see the stark differences in an application’s personality by comparing the Nike + screen and that of the Garmin Training Center.  It should be said that Nike’s intentions are likely different than those of Garmin’s, after all, as this web site is clearly a vehicle to drive their following to purchase new products.

It may also seem unfair to compare a Nike product, which no doubt has been combed over many times by their marketing and interactive agencies, to Garmin’s software that some may view almost as an afterthought to the device itself.

But simply seeing these screens together shows just how little attention or thought was given to the Garmin interface, even down to the decision to show the user’s birthday, weight, and name as primary information, leaving the workouts on the left as secondary.

Individual Workouts

Each of the GPS-enabled applications incorporate some map visualization into the interface, though some implementations are more crude than others.  For example, the Trail Runner system uses Google Maps, but the Garmin system appears to be a custom-built mapping engine, and looks like it belongs back in 1985.

Trail Runner:

The Trail Runner interface uses the Aqua interface components to deliver a consistent experience with the Mac user’s other applications.

This interface helps define the distinction between a good personality and a usable product.   It certainly looks the best, using the aforementioned Google Maps API, icons instead of burying everything in pulldown controls, and using color to connote similarities and differences between data.  Unfortunately, there are numerous usability problems at the task and process level, and not necessarily as a result of poor interface design.

That being said, an adaptable interface would be a huge blessing to the overall experience.  Though I will likely never go back to using the Nike + system, or use my iPhone instead of my Forerunner, these icons will stay cluttered at the top of the primary navigation as long as I use the software.  More advanced, or niche features, such as saving routes, managing my exercise plans and more may be better served for someone who uses the application for more than tracking runs, calories, and how frequently I exercise.

Perhaps crystallizing the argument that a good user experience can almost trump a less useful application, the personality of the Trail Runner application keeps me wanting to return to it, even though the system’s shortcomings continue to frustrate me.

Garmin Training Center

These screenshots from both Vista and Mac OS show that the bare, utilititarian application addresses the simplest use case: it displays how far I ran, where I went, and it shows my heart rate throughout the activity.

Vista top |  Mac OS below

Where is its personality? Good question.  The system buttons, controls, and components don’t tell any more of a story than a balance sheet. But the interface stays out of my way, unlike the more attractive Trail Runner.  In fact, if I didn’t know that this interface represented a person’s run, I’d probably assume it was graphing some complex engineering system.

Nike +

The Nike interface simply shows what data it has to work with: time and distance, and the speed over which that distance was traveled.

The bumps in the line indicate when the runner’s speed increases, and does not represent moving uphill or downhill as could be inferred by the running animation.

But even with the broken mental model of running uphill as an increase in speed, the interface still features more polish than the more advanced Garmin Training Center, and summarizes the information more prominently than the Trail Runner.

The soft shadows, gradients, and reflections all suggest a refined, sophisticated experience.  It’s also similar to many of Apple’s design assets, which strengthens the holistic bond between the shoe, pedometer, and iPod. And even though some of us in the design community may be tired of this look, it’s likely still perceived by their wider users as professional and accessible.

SportTracks

The SportTracks workout interface provides a unique approach compared to the earlier interfaces discussed here.

The personality is certainly down to business moreso than the Nike + system; the rows of data from prior workouts in the Daily Activity panel provide more context to the most recent workout (found in the Summary panel).  Some could argue that the interface doesn’t prioritize data effectively: quickly scanning the screen may not indicate where I should focus or what’s most important.   But perhaps that lack of heirarchy is part of the application’s personality: it’s designed to be put to work but flexible in how you use its features.

Workout Summary Screens

The summary screens don’t deviate from the personalities established by the screens discussed earlier.

The Garmin Training Center elects to display only numeric totals of distance and elapsed time in the top of table columns, whereas the Nike + system provides no numeric value of how far the user has run.

It’s unfortunate the Garmin Training Center doesn’t borrow from the SportTracks system and display a chart of my long runs compared to my tempo runs or cycling workouts; instead I get a crude map of Central America, central Africa and southern India.

The SportTracks (below)  summary interface maintains it’s rather bland personality while still keeping its data front and center.   I’m curious if additional data such as heart rate could have been represented using color in each bar representation of a workout.

Conclusion

These applications are proof that an interface’s personality can enhance the user experience. However, relying on such personality to compensate for usability shortcomings or limited features is difficult at best.

Determining what tasks users will want and use and the various mental models of how those users will engage with the system’s features are far more critical than applying gradients to backgrounds.  But when those backgrounds, in addition to layout, color, and language, cohesively align with how the user expects the web site or application to behave, a personality can develop that the satisfied user will turn to time and again.

Regardless if the issues are about government bailouts, the college football BCS, or if user interface design and interaction design are one and the same, a critical thinker identifies and reacts against twisted statistics or flawed logic.

However, it’s difficult to recognize fishy data published online by unfamiliar albeit seemingly reputable institutions. Even more challenging is understanding the value of their conclusions when these reports are cited by others, referenced in research, or included in legitimate press releases.

Self-Promotional Usability Marketing Reports

For example, the Australian business technology web site iTnews recently published “E-Commerce Sites Failing on Simple Usability“, which describes the results of testing 3,600 volunteers in Europe across 14 different web sites.  The article summarizes that “the average success rate for each task among UK users was 52 percent at best, and that the figure fell to around a third for some sites”.

Unfortunately, we’re not privy to what those tasks were, save for a brief mention of when shipping costs are provided and how to return damaged goods.

But it’s important to realize that returning damaged goods is a niche, fringe use-case.  And without knowing what is exactly a failure, we cannot determine if the user simply made a few wrong turns but ultimately found his way, or  if he threw the computer out the nearest window in frustration.

David Hamill of Good Usability also expresses his skepticism with the assigned tasks, commenting that  “somehow I doubt it was ‘Buy an iPod on Amazon’.

Most importantly, we can’t even be sure of the integrity of the research.  Since a usability company partnered with a research firm, and was not commissioned by one particular company to provide in-depth analysis, we can skeptically presume this report is just to publicize each both names to drum up future business.

It’s easy to believe a scenario where an unscrupulous company could provide a web usability test designed to stump volunteers, then in turn go to that company whose site just “failed” their test and offer to provide recommendations to increase their site’s usability.  Is this the case here? I don’t know, but unfortunately we can’t ignore the possibility.

Web Site Rankings

Many web site top-ten lists select “winners” based on criteria that’s either subjective or based on irrelevant data.

For example, the “Webometrics Ranking of World Universities” is a project by the Cybermetrics Lab, a research group belonging to the Consejo Superior de Investigaciones Científicas (CSIC), the largest public research body in Spain.

The study’s mission is to measure

the activity and visibility of the institutions and it is a good indicator of impact and prestige of universities. Rank summarizes the global performance of the University, provides information for candidate students and scholars, and reflects the commitment to the dissemination of scientific knowledge.

Though I’m clearly not in a position to question the quality of Spain’s largest public research consortium, we should read this particular report with a skeptic eye. Even this particular study’s mission statement is enough to alarm the casual reader.  Can a web site adequately indicate the “prestige” of its university?

And while I understand this study originated in Spain and there could be some meaning lost in the translation to English, I’m left to assume the rank “summarizes the global performance of the [University's]” web site and not the University itself.

If not, then the folks at Yale (39), Duke (33), and NYU (28) were bested by the likes of my own alma mater, the University of Maryland (at 19 and one step ahead of the University of Chicago, 20).

Even worse is when we dig deeper into how these results were founded:

Four indicators were obtained from the quantitative results provided by the main search engines as follows:

Size (S). Number of pages recovered from four engines: Google, Yahoo, Live Search and Exalead.

Visibility (V). The total number of unique external links received (inlinks) by a site can be only confidently obtained from Yahoo Search, Live Search and Exalead.

Rich Files (R). After evaluation of their relevance to academic and publication activities and considering the volume of the different file formats, the following were selected: Adobe Acrobat (.pdf), Adobe PostScript (.ps), Microsoft Word (.doc) and Microsoft Powerpoint (.ppt). These data were extracted using Google, Yahoo Search, Live Search and Exalead.

Scholar (Sc). Google Scholar provides the number of papers and citations for each academic domain. These results from the Scholar database represent papers, reports and other academic items.

The problem, of course, is that we cannot evaluate the scholarly value of a document based only on Google Scholar or simply count how many PowerPoint presentations exist on a web server to infer said “global performance”.

Sheer quantity of pages is also irrelevant, since we do not know how many of these pages are scholarly research and how many pages are sports highlights from a 1993 fraternity-league flag football game.

Gauging ranked universities’ reactions to this report is also worth monitoring.

For instance, MIT, the list’s #1 university web site based on this ‘research’, published their top ranking in an internal staff newsletter in a brief blurb, whereas Stanford and Harvard apparently make no mention of the report at all, based on site searches (I’m such a sleuth. Sign me up for CSI-DC).

And what of my undergraduate institution, the home of the Terrapins? Well, they went the opposite route and published a press release, aptly titled UM Ranked Among Globe’s Most Cybersavvy Universities.