Conducting Usability Tests
Printed Page 348-353
Conducting Usability Tests
Usability testing draws on many of the same principles as usability evaluations. For example, in a test, you start by determining what you want to learn. You choose test participants carefully, and you repeat the test with many participants. You change the draft and retest with still more participants. You record what you have learned.
The big differences between usability evaluation and usability testing are that testing always involves real users (or people who match the characteristics of real users) carrying out real tasks, often takes place in a specialized lab, is recorded using more sophisticated media, and is documented in more formal reports that are distributed to more people.
This section covers four topics:
- the basic principles of usability testing
- preparing for a usability test
- conducting a usability test
- interpreting and reporting the data from a usability test
Three basic principles underlie usability testing:
- Usability testing permeates product development. Usability testing involves testing the document, site, or software rigorously and often to make sure it works and is easy to use. Prototypes, newly completed products, and products that have been in use for a while are all tested.
- Usability testing involves studying real users as they use the product. Unlike usability evaluations, which often involve experts, testing is done by real users, who can provide important information that experts cannot. Real users make mistakes that experts don’t make. One well-known example relates to computer software that included an error-recovery message that said, “Press Any Key to Continue.” The manufacturer received hundreds of calls from users who couldn’t find the “Any” key.
- Usability testing involves setting measurable goals and determining whether the product meets them. Usability testing involves determining, first, what the user is supposed to be able to do. For instance, in testing a wiki, the testers might decide that the user should be able to find the “Edit” function and then edit and save a sentence successfully in less than 30 seconds.
Usability testing requires careful planning. According to usability specialist Laurie Kantner (1994), planning accounts for one-half to three-quarters of the time devoted to testing. In planning a usability test, you must complete eight main tasks:
Read more about your audience’s needs in Ch. 5.
- Understand users’ needs. To understand users’ needs, companies conduct focus groups, test existing products, have experts review the product, and conduct on-site interviews and observations of real users in the workplace.
- Determine the purpose of the test. Testers can test an idea even before the product is designed, to see if people understand it and like it. Or they can test a prototype to see if it is easy to use, or a finished product to see if it needs any last-minute improvements.
- Staff the test team. Extensive programs in usability testing involve many specialists, each doing one job. Smaller programs involve only a handful of people, each doing many jobs. For instance, a testing team might include an SME on the product, who can suggest workarounds if necessary; a test administrator, who administers the test to participants; a note taker, who fills out the evaluation forms and records important comments users make; and a videographer, who operates the recording equipment.
- Set up the test environment. A basic environment includes a room for the test participant and another room for the test observers. Figure 13.3 presents a photograph of a usability lab.
For information about proposals, see Ch. 16.
- Develop a test plan. A test plan is a proposal requesting resources; it describes and justifies what the testers plan to do.
- Select participants. Testers recruit participants who match the profile of the intended users. Generally, it is best not to use company employees, who might know more about the product than a real user would.
- Prepare the test materials. Materials for most tests include legal forms for the user to complete, an orientation script to help the participant understand the purpose of the test, background questionnaires, instructions for the participant to follow, and a log in which testers record data during the test.
- Conduct a pilot test. A pilot test is a usability test for the usability test. A pilot test can uncover problems with the equipment; the document, site, or software being tested; the test materials; or the test design.
The testing team has to plan the test carefully and stay organized. Typically, the team creates a checklist and a schedule for the test day, specifying every task that every person, including the test participant, is to carry out. Conducting the test includes interacting with the test participant both during the formal test and later, during a debriefing session.
Interacting with the Test Participant Among the most popular techniques for eliciting information from a test participant is the think-aloud test, in which the participant says aloud what he or she is thinking while using a document or a website. Consider the earlier example of FloorTraxx software for designing custom floors. In planning to test the software, you would first create a set of tasks for the participant to carry out:
- Calculate the area of a floor.
- Calculate the number of tiles needed for a project.
- Estimate the amount of adhesive needed for a project.
- Generate the bill of materials needed for a project.
- Calculate the cost of materials and number of hours of labor for a project.
As the participant carries out each task, he or she thinks aloud about the process. Because this process might make the test participant feel awkward, the test administrator might demonstrate the process at the beginning of the session by thinking aloud while using one of the features on a cell phone or finding and using an app on a tablet.
While the test participant thinks aloud, a note taker records anything that is confusing and any point at which the test participant is not sure about what to do. If the test participant gets stuck, the administrator asks a leading question, such as “Where do you think that function might be located?” or “What did you expect to see when you clicked that link?” Questions should not take the user’s knowledge for granted or embarrass the test participant for failing a task. For example, “Why didn’t you click the Calculate button?” assumes that the user should have seen the button and should have known how to use it.
In addition, questions should not bias the test participant. When testers ask a participant a question, they should try not to reveal the answer they want. They should not say, “Well, that part of the test was pretty easy, wasn’t it?” Regardless of whether the participant thought it was simple or difficult, his or her impulse will be to answer yes. Usability specialists Joseph S. Dumas and Janice Redish recommend using neutral phrasing, such as “How was it performing that procedure?” or “Did you find that procedure easy or difficult?” (1999). In responding to questions, testers should be indirect. If the participant asks, “Should I press ‘Enter’ now?” they might respond, “Do you think you should?” or “I’d like to see you decide.”
To ensure that the test stays on schedule and is completed on time, the test administrator should set a time limit for each task. If the test participant cannot complete the task in the allotted time, the administrator should move on to the next task.
ETHICS NOTE
UNDERSTANDING THE ETHICS OF INFORMED CONSENT
For legal and ethical reasons, organizations that conduct usability testing—especially tests that involve recording the test participant’s behavior—abide by the principle of informed consent. Informed consent means that the organization fully informs the participant of the conditions under which the test will be held, as well as how the results of the test will be used. Only if the participant gives his or her consent, in writing, will the test occur.
When you obtain informed consent for tests that involve recording, be sure to do the following six things:
- Explain that the test participant can leave at any time and can report any discomfort to the testing team at any time, at which point the team will stop the test.
- Explain that a video camera will be used and, before the recording begins, ask for permission to record the test participant.
- Explain the purpose of the recording and the uses to which it will be put. If, for example, the recording might be used later in advertising, the test participant must be informed of this.
- Explain who will have access to the recording and where it might be shown. A participant might object to having the recording shown at a professional conference, for example.
- Explain how the test participant’s identity will be disguised—if at all—if the recording is shown publicly.
- Explain that the test participant will have the opportunity to hear or view the recording and then change his or her mind about how it might be used.
Debriefing the Test Participant After the test, testers usually have questions about the test participant’s actions. For this reason, they debrief the participant in an interview. The debriefing is critically important, for once the participant walks out the door, it is difficult and expensive to ask any further questions, and the participant likely will have forgotten the details. Consequently, the debriefing can take as long as the test itself did.
While the participant fills out a posttest questionnaire, the test team quickly looks through the data log and notes the most important areas to investigate. Their purpose in debriefing is to obtain as much information as possible about what occurred during the test; their purpose is not to think of ways of redesigning the product to prevent future problems. Usability specialists Jeffrey Rubin and Dana Chisnell (2008) suggest beginning the debriefing with a neutral question, such as “So, what did you think?” This kind of question encourages the participant to start off with an important suggestion or impression. During the debriefing session, testers probe high-level concerns before getting to the smaller details. They try not to get sidetracked by a minor problem.
After a usability test, testers have a great deal of data, including notes, questionnaires, and videos. Turning that data into useful information involves three steps:
- Tabulate the information. Testers gather all the information from the test, including performance measures, such as how long it took a participant to complete a task, and attitude measures, such as how easy the participant found the task.
- Analyze the information. Testers analyze the information, concentrating on the most-important problems revealed in the test and trying to determine the severity and the frequency of each one.
- Report the information. Writing a clear, comprehensive report often leads the testers to insights they might not have achieved otherwise.
Although usability testing might seem extremely expensive and difficult, testers who are methodical, open-minded, and curious about how people use their documents or websites find that it is the least-expensive and most-effective way to improve quality.