The proliferation of usability labs is a sign of success for the field of user-centered design. Whether it’s a low-rent lab comprised of a couple adjacent conference rooms, a video camera, and a television, or a fully decked-out space with remote-control cameras, two-way mirrors, an observation room, and bowls of M&Ms — more and more companies are investing in such set-ups. Conducting user tests in labs is probably the most common means of getting user input on projects.
That’s a shame, because standard user testing practice is remarkably out of sync with reality.
The methods of standard practice, such as having someone sit in front of a computer, engage in a set of scripted tasks, and “think aloud,” haven’t really changed since the early 1990s. That was a time when a minority of the population even had computers, and people used them for purposes of Calculation (math, spreadsheets, and so on), Creation (word processors, graphics programs), and Capture (data entry). Typically, only one “program” ran at a time, files were stored on floppy disks, and the computer wasn’t connected to a network. Users focused on a single task for many minutes, if not hours on end.
These days, our customers’ technological world is much more complex. The bulk of their time online is spent engaged in Consumption (browsing the Web, listening to music), and Communication (email, instant messaging), though they still Calculate, Create, and Capture. They have multiple applications open, and multiple windows within those applications. Thousands of files and emails fill their hard drives, and they’re managing multiple devices, from computers to mobile phones to iPods. Technologies have driven users to a point of extreme distraction — recent research has shown that workers are interrupted an average of every 11 minutes.
We need to practice research methods that accept this complexity, and take it into account. Monolithic solutions are giving way to smaller point solutions, people are saving their information in a variety of places (personal computer, websites and hosted applications, handheld devices, print-outs), and reliance on stored passwords and favorites is deepening. And yet, in this climate, we still invite folks into a foreign lab, to use a computer that isn’t theirs, to leave behind their files, papers, and Post-It Notes, and then ask them to engage in a scripted series of uninterrupted tasks.
No. We need to go to them. We need them to have the things that help them manage their burdened digital lives. We need their friends beckoning them on instant messenger. We need their colleagues walking up to their cubes and interrupting with questions. If we’re designing products for those hectic environments, then we need to do more to appreciate that context.
When conducting interface assessments, we’ve given up lab usability in favor of remote usability. Participants remain in their home or office, and with screen-sharing software we can observe them like we would in a lab. But, since they’re using their own setup, we can also see their actual bookmarks, what happens when they download files, interruptions from friends or colleagues, and other markers of a truer experience.
This stuff isn’t new — others have written about this approach. I’m simply perturbed by researchers’ hesitancy to adopt this technique.
You get everything you get from lab usability, plus a more realistic context, geographic freedom, and the ability to immediately intercept people who are actually coming to your site. What’s more, there are no travel costs, and you can offer lower incentives because the process is less intrusive. The only thing you lose is face-to-face time, and you know what? For usability testing, that amounts to very little.
Increasingly, we find that our projects warrant getting out of our office and into our users’ lives. The generic term for this practice is “field research,” and that can mean talking to people in their homes, observing work practices in an office, or conducting guerrilla research in a public place like a shopping mall.
Such deep work can lead to insights you simply won’t get any other way. I led a project for a well-known consumer brand that wanted to develop a Web application to help people remember important occasions. Such projects often involve developing a prototype and inviting typical users in to see if they can use it. Even though we had a set of well-defined requirements, we decided to visit 12 people in their homes to understand the context in which such a tool would be used. Our observations opened our eyes to the messy complexity of how people manage this information. One woman had dates and addresses spread across a week-at-a-glance planner, a church-published mailing list, a Palm, a Psion-like device, and, most impressively, an Access database she developed for her wedding. This led to a realization that the basic product concept was flawed and would likely see little uptake, but it also revealed new opportunities around craft and creativity that had been rejected at an earlier stage.
So field research potentially saved tens, if not hundreds, of thousands of dollars in unnecessary product development, and revealed an opportunity for revenue generation that had previously been dismissed.
But isn’t this expensive?
Time and cost are common reasons for not conducting field research; however, our practice shows that it’s only marginally more expensive than lab work — and the insights gained make it more than worth it. For the sake of argument, here’s some back-of-the-envelope math on conducting research with 12 participants.
| ||Lab Usability||Remote Usability||Field Research|
|Preparation (Plan, Protocol)|| 1-2 weeks||1-2 weeks||1-2 weeks|
|Recruiting||1-2 weeks||1-2 weeks||1-2 weeks|
|Conducting Interviews||3 days||3 days||4 days (5-6 with travel)|
|Analysis||1-2 weeks||1-2 weeks||1-2 weeks|
|Totals||4-7 weeks||4-7 weeks||4-7 weeks|
|Equipment||(Camera, tapes) $400-600||$50 (audio tapes)||$400-600|
|Travel||$0||$0||$0 - $3000 (Air, ground, lodging, meals)|
|Facilities||$0 - $1,000/day||$0||$0|
|Phone charges||$0||$200|| $0|
|Totals||$1000-4800||$850 - 1450||$1600 - 5400|
Get to It
I want to reiterate that nothing proposed here is new or revolutionary. In fact, you’d be hard-pressed to find a research technique more rudimentary than “watch people in their own environments.” But there’s this unfortunate inertia in design practice that leads to the continuing use of ill-suited methods. Shake off your complacency and engage with your users more honestly. You’ll be amazed at what you find.
Peter Merholz is the Director of Practice Development and a founding partner of Adaptive Path, the world’s premier user experience consulting company. He is an experienced information architect, writer, speaker, and leader in the field of user-experience design. Clients include Cathay Pacific, Yamaha, and the California Health Care Foundation.