Group Usability Study

Overview

In the third phase of the group capstone project, your group will conduct a usability study of a high-fidelity (i.e., computer-based) prototype of your design. You will formulate a list of representative tasks, and recruit three to five participants to perform those tasks as you videotape them. By analyzing the videotapes, you will identify any usability problems that occurred, and you will propose design changes to remedy those problems.

Note: Your group is invited to run your studies in the state-of-the-art usability lab in EME 228. In order to do this, you must attend one of the two orientation sessions scheduled for April 8 and 10, and you must reserve blocks of time to use the lab through the WebCT "EME 228 Schedule" link. This EME 228 Primer can be used as a reference for running your studies in EME 228.

Steps

1. Prepare study tasks. By now, you are well aware of the core tasks to be supported by your technology. Construct a list of at least five representative tasks that your study participants will perform with your prototype. It is important to set the stage for your participants by constructing a realistic (but probably fictitious) scenario in which they are performing the tasks. For example, if you were testing a walk-up-and-use information kiosk, you might describe the scene as follows. "It is Saturday morning, and you are your friends are ready to explore New York City. You take the bus to Fifth Avenue, and when you exit, you see an information kiosk adjacent to the bus stop. Eager to find your way to the Institute of Art, you approach the kiosk. . ." For each task in your test, you could build on this scenario.

2. Write study instructions. (Done in previous deliverable.) Before participants begin working on the tasks, you will need to have them read (or read aloud to them as they follow along) a set of instructions. These instructions tell participants what to expect, tell them how to "think aloud," and share any other information that is relevant to their participation. This template document provides a starting point for the instruction booklet you'll give to participants; it includes sample instructions, a placeholder for your scenario, placeholders for each task, and a sample exit interview. Feel free to modify it to meet your needs.

3. Create informed consent form. (Done in previous deliverable.) Before participants begin participating in the study, they must sign an informed consent form that explains their rights and responsibilities as a participant in your study. For legal and ethical reasons, participants must be informed that their participation is voluntary, and that they can withdraw from the study at any time. Use this informed consent form as a starting point for your informed consent form.

4. Create background questionnaire and exit questionnaire. (Done in previous deliverable.) It is important to get a sense of the experience and educational level of your participants. To do this, you'll need to have participants fill out a background questionnaire prior to their participation in the study. In addition, you'll want to elicit participants' subjective impressions of your prototype once they're done with the study. To collect quantitative data that will help you gauge your prototype with respect to the user experience goals you have established, your questionnaire should begin with a set of Likert scale questions. You are required to choose a subset of questions from the standardized QUIS questionnaire (available off of the WebCT site) for this purpose, but feel free to also include your own custom questions to get at your user experience goals directly. In addition, at the end of the questionnaire, it is common to include a set of free form questions (e.g., "What did you like and not like about the software?" "Can you imagine using this software in your job?" ) Be sure to include a copy of the screening and exit questionnaire you administered as an appendix to your usability study report.

5. Recruit three to five participants and run them through the study. (Five participants are highly recommended to increase the reliability of your study results.) Start each study session by having the participant read and sign your informed consent form. Then have the participant fill out the background questionnaire. Third, have the participant (a) read the instructions aloud, or (b) follow along as you read the instructions aloud. Fourth, turn on your video camera, or Camtasia Recorder, and have the participant work through the study tasks. (Note: Whatever recording method you use, be sure to record audio as well as video. If you're using Camtasia Recorder, you must be sure that the recorder is set up to record audio, and that a microphone is hooked up to the computer being used by your participants.) Fifth, have the participant complete the exit questionnaire. Finally, thank the participant for participating; if you agreed to give out an enticement (cookies, candy, etc.) as a "thank you" for participation, this would be the time to do so. Note that you should design your study session so that it lasts no more than one hour; 30 to 45 minutes ought to be sufficient in most cases.

6. Analyze your data. As you review your recording of each participant session (or as you observe each participant session), create a list of "critical incidents" that occur; organize these incidents into logical categories that correspond either to the task, or to components of your interface. This critical incident spreadsheet may help you to compile and organize your critical incidents (notice there's a separate sheet for each participant). Next, synthesize your critical incidents into usability problems. For each usability problem you identify, indicate the number of participants who encountered the problem, the counter location(s) at which the problem occurred in each session, the estimated severity and scope of the problem, and a possible design change to remedy the problem. This usability problem spreadsheet may help you to compile your usability problems. Use the following guidelines to assess severity and scope:
• Severity. Severity is an assessment of a problem's impact on user performance. The following scale is derived from Dumas and Redish (1993):
  • Severity 1 problems prevent users from completing a task. Participants give up after a few tries or they need a hint to continue. For example, users consistently select an incorrect dialog option and do not know what else to do.
  • Severity 2 problems create significant delay and frustration. Participants continue to get lost or to use inefficient methods to accomplish a goal. For example, the lack of feedback to users confirming what they have just done causes them to do the task over to make sure they did it correctly.
  • Severity 3 problems have a minor effect on usability. For example, an unusual term in a dialog causes users to hesitate for a moment before making the correct choice.
  • Improvements. While not problems per se, improvements will make the task even easier to perform or learn. The interface doesn't hamper users but there is something that could make it even better.
• Scope. Scope is an assessment of how frequently users will encounter a problem. The more users that a problem affects, wider its scope. The following scale is derived from Dumas and Redish (1993):
  • Scope 1 problems will affect almost all users.
  • Scope 2 problems will affect many users.
  • Scope 3 problems will affect few users.
    
    
7. Assess whether usability study results satisfy your usability and user experience goals. Compute the average time it took participants to complete each task, the number of errors made, the average questionnaire question ratings, and any other data relevant to gauging your prototype with respect to your original usability and user experience goals. In the "Overview" subsection of the "Findings" section, present this data in a table of the following form:

• Write up a usability report. I'd like you to write your report in the standard format used by many industrial usability labs. Your report will be written as a memo to me (your "boss"). This usability report template illustrates the required format; please use it as a starting point for your own report. Also consult this exemplary usability report, which received an A in a previous offering of this course.

  
  Notes:
  

  • I have interspersed comments in this report to highlight important features of the report, and to give you hints on how to write a good report. Be sure to read my comments, in addition to the report itself, before you write your own report. While you may use any of the existing prose in the usability report template verbatim and without attribution, you may not copy the prose of the exemplary report verbatim. I will treat such copying as plagarism, and grade it accordingly.
  • The requirements have changed since the exemplary usability report was written. In particular, this exemplary usability report does not include the table described in (7) above.
  • Your report must include the following as appendices: informed consent form, instructions, tasks, screening questionnaire results, exit questionnaire results, critical incidents (use template), and summary of usability problems (use template).

Deliverables

Write up and hand in the usability report your prepared in step 7 above. Your usability report will be graded according to the following assessment form; please look at it prior to completing your usability report.

One of your group members should hand in a hardcopy of your group's document to your instructor by the due date. Be sure that all group members' names clearly appear on the document. In addition, by the due date, you are required to hand in to your instructor a CD or USB drive (which will be promptly returned) with the following:

(a) your project report (in .doc or .pdf format)

(b) your working prototype (the executable file, along with any instructions necessary to install it), and

(c) your digitized videotapes.

Remember, a key goal of this course is to help you develop written communication skills. Make sure that your document is easy to read, and is in proper English. If English is not your first language, be sure to enlist an editor (who is a native speaker) before you hand in your report. This will give you a chance to improve your English, and will improve the general readability of your report for me and others who read the document on-line. I will mark you down if your document contains inordinate amounts of grammatical errors or incorrect sentence structures.