Читать книгу Remote Research - Tony Tulathimutte - Страница 3
ОглавлениеChapter 1
Why Remote Research?
What’s Remote Research GoodFor?
When to Use Which Remote Method
In-person lab research used to be the only game in town, and as with most industry practices, its procedures were developed, refined, and standardized, and then became entrenched in the corporate R&D product development cycle. Practically everything gets tested in a lab nowadays: commercial Web sites, professional and consumer software, even video games (see Figure 1.1).
Photo courtesy of Danny Hope
Figure 1.1 http://www.flickr.com/photos/rosenfeldmedia/4218821411/ Brighton University’s usability lab, from behind the traditional two-waymirror.
The Appeal of Lab Research
Part of the appeal of lab-based user research was that it provided a seemingly scientific basis for making decisions by using observational data, instead of someone’s error-prone gut instincts. Stakeholders appreciated the firm protocol and apparent reliability of properly managed lab research. Lots of user research practitioners continue to perform lab research just because it’s what people have been doing for a long time.
Market Research vs. User Experience Research
Let’s make something clear. Focus groups are practically synonymous with user research in most people’s minds, and focus groups belong to the world of market research. But there’s a huge difference between market research and user experience (UX) research. Market research is much more common and comprises the lion’s share of research spending; UX research comprises just a fraction of that (see Figure 1.2).
Figure 1.2 http://www.flickr.com/photos/rosenfeldmedia/ 4218822967/ The relationship between market research and UX research. The two fields seem similar, but they have different goals and take different forms.
However, this book is about user experience research, not market research. The main difference between the two fields is that market research focuses on opinions and preferences, whereas UX research focuses on behaviors. The distinction can be confusing, especially since a lot of online consumer research companies try to convince you they give you insight into “what your customers are doing on your Web site,” when they’re really just providing opinions.
A market research study might have goals like these:
“Determine how users respond to our branding.”
“Identify different segments’ color preferences for the homepage.”
“See if users like our new mascot.”
“Determine what users enjoy most and least about our site.”
While the goals of a UX study, on the other hand, would sound more like these:
“Can anyone actually use my interface?”
“Determine where users make errors in completing a purchase.”
“See whether users can successfully create a playlist.”
“Understand why users aren’t logging in.”
“See how users mentally organize different product categories.”
It’s important to keep in mind that market research is pretty useless over small sample sizes. Opinions can vary widely across demographics and location, are very sensitive to the phrasing of the research questions, and can change fairly quickly. Behavior, on the other hand, is fairly consistent across demographics and location for many tasks, and most usability flaws in a given software interface can be uncovered in a moderated study by a much smaller number of users.
How small? That’s a contentious question. UX luminary Jakob Nielsen (in)famously claimed that having five users was enough to uncover 80% of usability flaws in an interface, but others like Jared Spool insist that the number depends on factors such as user segmentation, risks associated with the errors, task complexity, and so on. At any rate, our point is that a moderated UX study usually requires much fewer people than a market research study.
Put it this way: ask 10 people what they think about how well a door is designed, and their comments might not overlap at all. One blames the condition of the hinges, another talks about the weight of the door, another complains about the color of the doorframe, and so on. But if you observe 10 people walking through the door, and the first two accidentally try to push when they ought to pull, then you’ve found your design flaw right there.
So, to put it all together: whether you go with market research or UX research depends on what you’re trying to find out. This book is about UX research, so it’s focused on user behaviors rather than opinions.
Further reading about the sample size question
Turner, C. W., Lewis, J. R., and Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084–3088). Boca Raton, FL: CRC Press.
Lewis, J. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. The International Journal of Human–Computer Interaction 13(4), 445–479.
Lindgaard, G., and Chattratichart, J. (2007). Usability testing: What have we overlooked? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, CA, USA, April 28–May 03, 2007). CHI ’07. ACM, New York, NY, 1415–142
Is Lab Research Dead?
Heck no. Lab and remote research share the same broad purpose: to understand how people interact and behave with the thing you’ve made (from here on, let’s just call it “the interface”). There’s no need to set up a false opposition between the two approaches—one isn’t inherently better than the other. Despite the versatility of remote research, there are lots of reasons you might want to conduct an in-person study instead, most of which have to do with security, equipment, or the type of interaction you want to have with your research participants. More generally, lab research is appropriate when you need a high degree of control over some aspect of the session, such as the following situations.
Info security. Security is often a concern for institutions like banks and hospitals, which deal in sensitive information, or companies concerned with guarding certain types of intellectual property. If you’re testing a top-secret prototype, you obviously don’t want to let people access something from their home computer, where it could be saved or screen-captured. On the other hand, you might also be doing a study on users who would be secretive about sharing what’s on their screen—government employees, doctors, or lab technicians, for instance. Either way, you’ll want to test users in a controlled lab environment to keep things confidential, especially if what you’re testing is so hush-hush that you must have your users sign a nondisclosure form.
Inability to use screen sharing. You might also want to use a lab if your users are unable to share their screen over the Internet, for whatever reason. Some studies (of rural users, cybercafe patrons, etc.) may require you to talk to users who don’t have reliable high-speed Internet connections, who own computers too slow or unstable to use screen sharing services effectively, or who have operating systems incompatible with the screen sharing tools you’re using. These restrictions apply only to moderated studies, for which you need to see what’s on your users’ screens.
The need for special equipment. Depending on the interface you’re testing, you may require certain special software or physical equipment to run the study properly, which is most often the case with software that’s still under development. Getting users to install and configure tools to run elaborate software can be a pain (though that’s not unheard of), and requiring users to have certain equipment can make recruiting needlessly difficult.
The importance of seeing the user’s body. Some kinds of research will require you to study certain things about the user that are difficult to gather remotely. UX research has recently begun using eye-tracking studies, and for that kind of study, you’d need to bring the users to the eye-tracking device. Other studies might require you to attend to the participants’ physical movements, which may be difficult to capture with a stationary webcam. And then there are multiuser testing sessions, in which a single research moderator facilitates many participants at once; screen sharing is currently not well suited to sharing multiple desktops at once, though some tools (e.g., GoToMeeting) make it relatively painless to switch from one desktop to another. We want to emphasize, however, that for most studies, seeing the user at all is not actually important; we explain why in Chapters 5, “Moderating,” (see “Ain’t Nothing Wrong with Using the Phone”) and 10, “The Challenges of Remote Testing.”
Although these situations are all compelling reasons to conduct in-person research, part of what we want to demonstrate in this book is that remote research is very broad and adaptable, and even if a study is conducted in a lab, elements of remote methods can be adapted and incorporated to enhance in-person research methods. We’ll get to that in Chapter9, “New Approaches to User Research.”
Why I Went Remote
by Brian Beaver
Old habits die hard, but for any number of reasons—cost, convenience, international testing—a handful of former lab researchers have switched to remote methods and never looked back. Brian Beaver, award-winning creative director at Sony, explains why he decided to go with remote methods after many years of lab testing.
On Going Remote
I have quite a bit of experience, either organizing user research sessions or participating in them, especially at Sony. My research has been pretty varied and the outcomes are always interesting, but I’m a big fan of remote usability testing. It seems to give me the best bang for my buck.
I’d read about it a few years back and had done some work with Adaptive Path when it had a focus on usability. Adaptive Path referred me to Nate [Bolt, coauthor of this book], and when he shared his remote approach with me, I knew we were in sync because a lot of the pain points and skeptical raised eyebrows around results we’d obtained in previous lab testing instantly diminished with remote usability testing.
The pain points always involved recruiting. With a Web site you have such a diverse geographic base that it can be challenging to bring a core group of your users together in one location. Sony tends to be very protective of its customer information, and wouldn’t share it with a research company for the purposes of recruiting, so we’d have to take on that task ourselves, which was always painful.
The raised eyebrows were always about participant motivation and validity of the recruiting process and methodology. There were always questions: How valid are these findings? Are these real users? But when you’re intercepting users who are on your Web site in the middle of performing a task, those questions evaporate.
On Participating in Remote Research
In the past we’d invite our business partners or stakeholders to the lab, but it was difficult to get them to take time out of their day to travel to the lab, and it was a big production. But if they can just bring their laptop to a conference room down the hall and just be there to listen in, it’s fantastic. You’d get the same advantages if you had everyone available to go to the lab testing, and the level of engagement is a lot greater. By having a lot of stakeholders in the room, you get more diverse viewpoints, and the interaction between us observers and the moderator tends to be lively—we chat throughout that whole interview. The ability to observe and discuss things as they come up and then immediately give feedback to the moderator is really powerful.
Because we are involved in the research process, we’ve got our customers and the usability to consider on the one hand, and on the other hand we have a lot of business stakeholders who have strong opinions about how things should be done. So having everyone in the room watching the feedback and engaging with the process is really powerful. We were recently in the middle of a digital camera usability session and were asking the user to go through the features and content we have on the site, and the customer’s going through it and he’s like, “This all seems really impressive, but I really just want to know if it takes great pictures.” And you see this light bulb go off above the product marketing people’s heads. We’re so close to this that we have absolute myopia. It was a real eye-opening moment.
On Benefiting from Remote Methods
One study was about TVs and the TV shopping process. Sony has a broad line of TVs, somewhere around 9 to 10 different series, and each has a dozen size options, so you have a lot of choices. During the study there was an “Ah-ha!” moment, a phenomenon we haven’t seen before: people would often have half a dozen to a dozen sites already open when we contacted them, and they were seamlessly going between sites like Engadget, CNET, Sony, Circuit City, Best Buy, really taking advantage of browser tabs to cross-shop and gather information. We simply wouldn’t have gotten that insight from a lab environment because we wouldn’t have been intercepting people in their natural browsing environment; instead, they’d sit down, have the browser already open, and they’d go. So that behavior would have been completely missed.
The outcome was that, knowing that customers are looking not only for customer reviews but trustworthy, third-party editorial content, we’re actively pursuing ways to bring that content into the SonyStyle site, so that from within the interface they can access that info, instead of relying on the multi-tab approach. In the past, if a product was awarded an editor’s choice, we would have put that on the page as a badge of honor, but I doubt that we would have ever actually included the editorial alongside the product, if it hadn’t been for this study.
Advice for Those Considering Going Remote
If we’re talking about remote testing for Web sites, from my perspective it’s really a nonchoice. Having the benefit of intercepting users that are already coming to your site in order to perform a task already puts you so far ahead of the game because the motivation is there, you’ve got them captive, and you just gain so many more insights compared to creating an artificial environment with artificial motives. So you know from the quality and granularity of the results you’re going to get, it’s so much richer. If given the choice, I’ll never go back to lab testing again. And there’s the cost savings. Clearly, overall, it’s a less costly proposition. You avoid all the travel costs. There’s always a dud user in every batch of lab participants, and the great thing with usability testing is, if you start talking to someone you want to cut loose, it’s no harm; you can move on to the next person, as the recruiting form is literally filling up before your eyes.
What’s Remote Research GoodFor?
Again, most studies can successfully be done either in person or remotely, but, just as there are times when lab testing is more appropriate, there are also times when it makes more sense to use remote research methods.
Time-Aware Research
Remote research is more appropriate when you want to watch people performing real tasks, rather than tasks you assign to them. The soul of remote research is that it lets you conduct what we call Time-Aware Research (TAR). By now, UX researchers are familiar with the importance of understanding the usage context of an interface—the physical environment where people are normally using an interface. Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest. This is possible because of live recruiting (the subject of Chapter 3), a method that allows you to instantly recruit people who are right in the middle of performing the task you’re interested in, using anything from the Web to text messages. Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.
Consider the difference between these two scenarios:
You’ve been recruited for some sort of computer study. The moderator shows you this online map Web app you’ve never heard of and asks you to use it to find some random place you’ve never heard of. This task is a little tricky, but since you’re sitting in this quiet lab and focusing—and you can’t collect your incentive check and leave until you finish—you figure it out eventually. Not so bad.
You’ve been planning a family vacation for months, but you’ve been busy at work so you procrastinated a bit on the planning, and now it’s the morning of the trip and you’re trying to quickly print out directions between finishing your packing and getting your kids packed. Your coworker told you about this MapTool Web site you’ve never used before, so you decide to give it a shot, and it’s not so bad—that is, until you get stuck because you can’t find the freaking button to print out the directions, and you’re supposed to leave in an hour, but you can’t until you print these damn directions, but your kids are jumping up and down on their suitcases and asking you where everything is. Why can’t they just make this stupid crap easy to use? Isn’t it obvious what’s wrong with it? Haven’t they ever seen a real person use it before?
Circumstances matter a lot in user research, and someone who’s using an interface in real life, for real purposes, is going to behave a lot differently—and give more accurate feedback—than someone who’s just being told to accomplish some little task to be able to collect an incentive check. Time-awareness is an important concept, so we’ll bring it up again throughout this book to demonstrate how the concept relates to different aspects of the remote research process (recruiting, moderating, and so on).
Note
TAR Keeps You in the Right 1985
Remember that diagram in Back to The Future II? Doc argues that messing with time has sent the world crashing hopelessly toward an alternate reality where things are horrible—the “wrong 1985.” And that’s sort of what happens when you try to assign people a hypothetical task to do at a time when they may or may not actually want to do it: you’re meddling with their time, and it’ll create results that look like the real thing but are all wrong.
When you schedule participants in advance and then ask them to pretend to care, you’re sending your research into the wrong 1985. If you don’t want to create a time paradox—thereby ending the universe—you should do time-aware research.
Other Benefits of Remote Research
Here are some additional benefits of remote research.
Geographic diversity. Even if you do have a lab, the users you want to talk to may not be able to get to it. This is actually the most common scenario: your interface, like most, is designed to be accessed and used all around the world, and you want to talk to users from around the world to get a range of perspectives. Will Chinese players like my video game? Is my online map widget intuitive even for users outside Silicon Valley? Big companies like Nokia and Microsoft are often able to conduct huge, ambitious research projects to address these questions, coordinating research projects in different labs around the world, flying researchers around in first-class. If you don’t have the cash for an international Gorillas-in-the-Mist project, then remote research is a
no-brainer solution. If you can’t get to where your users are, test them remotely.
Ability to test almost anywhere. Remote research has comparatively minimal setup requirements and can reach anywhere that computers and the Internet can go: you can be anywhere; your participants can be anywhere. Lone-wolf consultants and start-up teams working out of cafés can have trouble finding the quiet office space they need to do in-person testing. If it’s too much bother to set up a proper lab, go remote; all you’ll need is a desk.
Some reduced costs. Beyond travel expenses, other costs associated with lab testing may be reduced or eliminated when you test remotely. With live recruiting methods, you can get around third-party recruiting costs, and because the recruiting pool is larger, you may not have to offer as much in the way of incentives as you might otherwise to attract enough participants. Because sessions are conducted through the computer, you can use relatively inexpensive software to replace costly testing accessories, such as video cameras, observation monitors, and screen recording devices. (Note, however, that the overall cost of a remote research study is often comparable to an in-person study for many reasons; see Chapter 10 for reasons why.)
Quicker setup. Closely related to the issue of money, as always, is time. Nearly all existing recruiting methods take many weeks. Recruiting agencies usually require a couple of weeks to gather recruits, and writing out precise recruiting requirements and explaining the study to them can eat up a lot of time. Getting users from your own mailing list can be faster and moderately effective, but what if you don’t have one? Or what if you’ve overfished the list from previous studies, or you don’t want to spam your customers, or you’re looking to test people who’ve never used your interface or heard of your company before? In any of these cases, recruiting your users online makes a lot of sense, since it allows you to do your recruiting as research sessions are ongoing. (We teach you how to do all this in Chapter 3.)
Context-dependent interfaces. Some interfaces just don’t make any sense to test outside their intended usage environment. If you need users to have their own photos and videos to use in a video editing tool, having them bring their laptop or media to a lab will be a tremendous hassle. Or, let’s say you’re testing a recipe Web site that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment.
When to Go Remote
If you have the gumption, you can test almost anything remotely. There are ways to get around nearly any obstacle, but the approach you take is all about what’s most practical and accurate. If it’s significantly cheaper, faster, or less of a hassle for you to just bring people into a lab, then by all means bring ’em in. Sometimes this decision can be a tough call; users in the developing world may have limited access to the Internet, for instance, so you’d have to decide whether it’s worthwhile to fly over and talk to users in person, or to find people from that demographic in your area, or to arrange for the users to be at a workable Internet kiosk to test them remotely.
For clarity’s sake, let’s talk about some clear-cut cases of things you should and shouldn’t test remotely.
Remote testing is a no-brainer for Web sites, software, or anything that runs on a desktop computer—this is the kind of stuff remote research was practically invented to test. The only hitch is that the participants need to be able to use their own computer to access whatever’s being tested. Other Web sites besides your own are a cinch: just tell your users during the session to point their Web browsers to any address you want. If you’re testing prototype software, there needs to be a secure way to digitally deliver it to them; if it’s a prototype Web site, give them temporary password-protected access. If the testing is just too confidential to give them direct access on their computer, you can host the prototype on your own computer and use remote access software like VNC or GoToMeeting to let them have control over the computer. There’s almost always a way to do it.
The stuff you test doesn’t even have to be strictly functional. Wireframes, design comps, and static images are all doable; we’ve even tested drawings on napkins (really). Just scan them in to a standard image format and put them on a Web site. Make sure the user’s browser doesn’t automatically resize it by using a plain HTML wrapper around each image. There are also plenty of software solutions (like Axure and Fireworks) that can help you convert your images to HTML.
Can you test programs that require users to enter personal information? Yes, but make sure to give your participants a way to enter “dummy” information wherever they’re required to enter sensitive or personally identifying information. (According to Rolf Molich, people act differently when using dummy information, so bear that in mind.) If you require the participant to use real personal information, be sure to obtain explicit consent right at the beginning of the testing session (an issue covered in Chapter 4); you don’t want to spend 20 minutes on the phone with a user only to have to terminate the study over privacy issues.
Most remote research tools (screen sharing, recording, chat, etc.) are suited for a computer desktop environment, so physical products are harder to test remotely. We’re just beginning to see mobile device and mobile interface research become feasible, and we’ve researched interfaces like cars and computer games using some remote research methods. Plus, webcams, Web video streaming, and wireless broadband are all becoming more accessible, so there’s plenty of hope. But physical interfaces will require you to come up with some creative solutions and workarounds beyond the standard remote desktop testing approach. These approaches may or may not be worthwhile; see Chapter 9 for examples of some of these alternative remote approaches.
Case Study: Lab vs. Remote
By Julia Houck-Whitaker, Adaptive Path
(and Bolt | Peters alum)
In 2002, Bolt | Peters conducted two remote studies on the corporate Web site of a Fortune 1000 software company. Both studies used identical test plans, but one was executed in a traditional usability lab, whereas the other was conducted remotely using an online screen sharing tool.
Summary
Our comparison showed key differences in the areas of time, recruiting, and resource requirements, as well as the ability to test geographically distributed user audiences. Table 1.1 summarizes the key differences we found comparing the two methods. There appeared to be no significant differences in the quality and quantity of usability findings between remote and in-lab approaches.
Table 1.1 Overview Comparison of Lab and Remote Methods
http://www.flickr.com/photos/rosenfeldmedia/4286397757/
Detailed Comparison of Methods
Tables 1.2, 1.3, and 1.4 break down the process for each of the recruiting, testing, and analysis phases, respectively. The left-hand column describes the lab study details; the right-hand column describes the remote study details.
Table 1.2 Lab vs. Remote Recruiting
http://www.flickr.com/photos/rosenfeldmedia/4287138014/
Recruiting for the lab-based study was outsourced to a professional recruiting agency (see Table 1.3). Ten users were recruited, screened, and scheduled by G Focus Groups in San Francisco, including two extra recruits in case of no-shows. Recruiting eight users through the recruiting agency took 12 days. Agency-assisted recruiting successfully provided seven test subjects for the lab study; the eighth recruit did not fulfill the testing criteria.
Recruiting for the remote study was conducted using an online pop-up from the software company’s corporate Web site. The recruiting pop-up, hosted by the researchers, used the same questions as the G Focus Groups’ recruiting screener. Users in both studies were selected based on detailed criteria such as job title and annual company revenues. Respondents to the online screener who met the study’s qualifications were contacted in real time by the research moderators. The online recruiting method took one day and yielded eight users total from California, Utah, New York, and Oregon. Normally, the live screener requires four days of lead time to set up, but in this case it was completed for a previous project so setup was not necessary.
Table 1.3 Lab vs. Remote Environment
http://www.flickr.com/photos/rosenfeldmedia/4287138116/
The lab study was also conducted from the software company’s in-house usability lab. The recruits for the lab study went to the lab in Pleasanton, California to participate and used a Windows PC. In addition to users’ audio and screen movement capture, users’ facial expressions were also recorded.
The remote usability study was conducted using a portable lab from the software company’s headquarters in Pleasanton, California. The live recruits participated from their native environments and logged on to an online meeting allowing the moderators to view the participants’ screen movements. The users’ audio and screen movements were captured.
The lab study uncovered similar issues of similar quality and usefulness to the client when compared with the remote study results (see Table 1.4). The remote study uncovered usability issues of high value to the client. The lab method uncovered 98 key findings, compared with 116 findings in the remote study (not a statistically significant difference).
Table 1.4 Lab vs. Remote Findings
http://www.flickr.com/photos/rosenfeldmedia/4286397971/
The Case Against Remote Research
by Andy Budd
Even though we’re obviously firm advocates of remote methods, not all UX practitioners agree. Andy Budd is the creative director at Clearleft, a renowned London-based team of UX and Web design experts. Clearleft are the makers of Silverback, in-person usability testing software for interface designers and developers. Andy isn’t such a big fan of remote methods for moderated studies—here’s why.
Full disclosure: I’ve done very little remote testing, and the reason is that I’ve never found a credible need to do it. We’ve always found other ways of testing that didn’t require a remote approach. My issue is less about the negatives of remote testing and more about the positives of in-person testing.
Now, that’s not to say we test in a lab; I find that labs give a veneer of formality and scientific accuracy to studies, which they often don’t have. And testing labs and equipment are often more expensive, and tend to bog things down. So we take a grittier approach—just a meeting room and a video camera or some screen capture software.
We gain a lot of information by being in the room with people. They say 90% of communication is nonverbal. It’s about the cues in people’s tone of voice or posture. When you’re with a test subject, you pick up these signals more easily. With online video conferencing such as Skype, social conventions break down; you’re not able to read the cues that tell you when one person stopped talking or when it’s OK for another person to start talking. You get lag, and people talk over each other. Communicating remotely is difficult to do well, and it’s possible that it’s to do with our ability to use these new tools and technology. I wouldn’t be surprised—give it 20 or 30 years—when video conferencing becomes a norm and we’ve learned how to understand and read these subtle cues better. For now, there’s the potential to lose 90% of the information that’s coming through to you if you’re not testing in person.
On the Shortcomings of Remote Methods
Usability testing is all about empathy. It’s about creating a connection. That kind of empathy is difficult to create through Web conferencing, and it’s that gulf of miscommunication that makes it less attractive to me. I think there are instances where you should use remote moderated testing, often when it’s impossible to recruit users to a specific location. Recently, we were working on a project for a South American site. We wanted to speak to Brazilians, so initially we thought to do some kind of remote testing, but then realized that there was actually a large student and ex-pat Brazilian community here [in Brighton]. So instead we went to a Brazilian café and sat down and just chatted to actual Brazilian people who happened to be living in the UK. Some people said, “How can you possibly say that talking to Brazilian people in a Brighton café is exactly the same as in a favela in São Paulo?” But again, the difference is so subtle as to make little difference on the results, particularly when testing in small numbers. Sure, if you’re doing a scientific test and looking across very large sample sets, these minute effects are going to play a bigger role.
It also depends on what you are testing. There are some obvious cultural differences with the way people use the Web, but there are also universal habits: registering for a service, noticing positioning, etc. It’s unlikely that the Brazilian community in Brighton is not going to pick up something the community in Sao Paulo might.
On the Use of Technology in UX Research
People often try and find technological answers to human problems. A lot of the drive for remote testing is an attempt to find shortcuts. “Oh, it’s difficult to find test subjects, so let’s get technology to help us out.” Today I was reading an article where someone was saying, “How can we do remote ethnographic research? Can we get live cameras streaming?” And I thought, “Do a diary study.” Diary studies are probably the ultimate in remote ethnographic research. You don’t need a webcam streaming back to Mission HQ. People love tinkering with technology because it makes them feel like superheroes; it’s something to show off. I believe in human solutions; I think technology is often used as a crutch.
I think remote testing is still in its infancy. It’s based on the technology that’s available. It’s preferable to have lightweight technology that you can send to a novice user, double-click on it, and it opens and installs. But if you’re looking at an average computer desktop, which is now above 1024×768, and you also want to capture the person’s reactions, you want to send audio and video down that pipe as well—that’s a complicated problem. You need good bandwidth to do this really well. So then you create artificial problems, because you’re limited to people who have got pretty good tech and bandwidth. And so that would probably prevent us going and doing remote testing with somebody in a cybercafé in Brazil.
What I am kind of interested in is unmoderated [i.e., automated] remote testing, because it’s a hybrid between usability testing and statistical analysis or analytics. The benefit is that you can test on a much wider sample set. It complements in-person usability testing.
On the Purpose of User Testing
The point is to develop a core empathetic understanding of what your users’ needs and requirements are, to get inside the heads of your users. And I think the only way you can do that is through qualitative, observational usability testing. There are lots of quantitative tools out there, stuff that can tell you what’s happening, but it can’t necessarily tell you why it’s happening. We’ve all done usability tests where you watch people struggle and have a real problem doing something, and you can see they’re having a problem. Then they’ll go, “Oh, yes. It was easy.” We did a test where a user thought he’d purchased a ticket, and he hadn’t, but he’d left thinking he had. If he’d told you, “Yes, I’ve succeeded,” you would have been mistaken. Watching and observing what users do is very enlightening. Frankly, it’s easier for people to learn from direct experience than through analyzing statistics. There’s nothing like actually watching people and being in the same room.
Moderated vs. Automated
So, you’ve decided it’s worth a shot to try a remote research study. Feels good, doesn’t it? The first thing to know is that remote research can be roughly divided into two very different categories: moderated and automated research.
Figure 1.3 http://www.flickr.com/photos/rosenfeldmedia/4218824495/ Moderated research: a researcher (“moderator”) observes and speaks to a participant in another location. Outside observers can watch the session from yet a third location and communicate with the moderator as the session is ongoing.
In moderated research, a moderator (aka “facilitator”) speaks directly to the research participants (see Figure 1.3). One-on-one interviews, ethnographies, and group discussions (including the infamous focus group) are all forms of moderated research. All the parties involved in the study—researchers, participants, and observers—are in attendance at the same time, which is why moderated research is also sometimes known as “synchronous” research. Moderated research allows you to gather in-depth qualitative feedback: behavior, tone-of-voice, task and time context, and so on. Moderators can probe at new subjects as they arise over the course of a session, which makes the scope of the research more flexible and enables the researcher to explore behaviors that were unforeseen during the planning phases of the study. Researchers should pay close attention to these “emerging topics,” since they often identify issues that were overlooked during the planning of the study.
Automated (or “unmoderated”) research is the flip side of moderated research: the researcher has no direct contact or communication with the participant and instead uses some kind of tool or service to gather feedback or record user behaviors automatically (see Figure1.4). Typically, automated research is used to gather quantitative feedback from a large sample, often a hundred or more. There’s all sorts of feedback you can get this way: users’ subjective opinions and responses to your site, user clicking behavior, task completion rates, how users categorize elements on your site, and even your users’ behavior on competitors’ Web sites. In contrast to moderated research, automated research is usually done asynchronously: first, the researcher designs and initiates the study; then the participants perform the tasks; then, once all the participants have completed the tasks, the researcher gathers and analyzes the data.
Figure 1.4 http://www.flickr.com/photos/rosenfeldmedia/4218825627/ Automated research: a Web tool or service automatically prompts participants to perform tasks. The outcome is recorded and analyzedlater.
When to Use Which Remote Method
There’s plenty of overlap between automated and moderated methods, but Table 1.5 shows how it generally breaks down.
Table 1.5 Moderated vs. Automated Research http://www.flickr.com/photos/rosenfeldmedia/4286398073/
Moderated research is qualitative; it allows you to observe how people use interfaces directly. You’ll want a moderated approach when testing an interface with many functions (Photoshop, most homepages) or a process with no rigid flow of tasks (browsing on Amazon, searching on Google) over a small pool of users. Since it provides lots of context and insight into exactly what users are doing and why, moderated methods are good for “formative research” when you’re looking for new ideas to come from behavioral observation. Moderated research can also be used to find usability flaws in an interface. We cover the nitty-gritty of remote moderated research in Chapter 5.
Automated research is nearly always quantitative and is good at addressing more specific questions (“What percentage of users can successfully log in?” “How long does it take for users to find the product they’re looking for?”), or measuring how users perform on a few simple tasks over a large sample. If all you need is raw performance data, and not why users behave the way they do, then automated testing is for you. (Suppose you just want to determine what color your text links should be: testing every different shade on a large sample size to see which performs best makes more sense than closely watching eight users use three different shades.) Also, some automated tools can be used to gather opinion-based market research data as well, so if you’re looking for both opinion-based and behavioral data, you can often gather both in a single study. And certain conceptual UX tasks, like card sorting and A/B testing, are well supported by automated tools. See Chapter 6 for a more thorough look at the various automated research methods.
You don’t necessarily have to choose between moderated and automated testing, or even between lab and remote methods. You can even conduct multiple studies on the same interface, using the findings from one study to add nuance to another. That’s probably excessive for the average study, but for really large-scale projects where you just want to gather every bit of information you can (a new version of a complex software program, an overhauled IA, etc.), being comprehensive can’t hurt.
After reading this chapter, you should have a good idea of whether or not remote research suits you. Give it a try—if it’s not your thing, you can always go back to lab testing. We won’t tell anyone.
Chapter Summary
Do a lab study when you need to use special equipment, keep the interface 100% secure, see the user’s physical movements, or when you can’t use screen sharing tools.
Remote research has its own strengths, the greatest of which is that it enables Time-Aware Research, in which you observe users performing tasks you’re interested in observing right at the moment they’d naturally perform them.
Remote methods can also give you greater user diversity, cut travel costs, allow you to test from anywhere, be quicker to set up, and can be used to test context-dependent interfaces that wouldn’t make sense in a lab.
Currently, remote methods work best for testing functional computer interfaces, prototypes, wireframes, design comps, and mockups. You can test physical products with the right setup, but it’s slightly harder.
There are two kinds of remote research: moderated and automated. Moderated research has the researcher communicating with and observing users as they perform tasks; automated research has researchers using online tools and services to collect behavioral data from users automatically.
Generally, moderated research is good for collecting rich, qualitative behavioral data from a small sample, while automated research is good for collecting quantitative data over a larger sample.
Time-Aware Research.
Recruit someone who’s in the middle of a task.
Observe their behavior.
