In-Class Activity #10:
User Study Analysis

Overview and Learning Objectives

In this In-Class Activity, you will gain hands-on experience with usability study analysis by analyzing a segment of a recording of an actual usability test.

The specific learning objectives of this in-class activity are:

• to develop an ability to identify a usability problem on a recording;
• to develop an ability to diagnose the cause of a usability problem; and
• to develop an ability to make design recommendations based on an analysis of a usability problem.

Notes:

• For this in-class activity, we will need to work with one partner on a laptop computer.
• Your instructor will give you a CD-ROM or USB drive from which you can obtain the video files.
• It is very important that you delete these files from your harddrive prior to leaving class today. Your keeping these video files would constitute a violation of the informed consent agreement under which these recordings were made. Thanks!
• Make sure that you and your partners bring a set of headphones to this lab, and listen to the videos through the headphones; your instructor will lend you an audio splitter so that both of you can listen to the audio at the same time.

Steps

For this in-class activity, you will work in pairs. Follow these steps:

1. Obtain a CD-ROM or USB drive containing a set of video files to review. Copy these files temporarily onto your laptop's hard drive, keeping in mind that you will be required to delete them before you leave class.
2. With your partner, launch Windows Media Player and carefully review at least two of the following video clips (all are about 10 minutes in length; your instructor will tell you which ones):
   • P4-Clip.wmv: 11:40 - 22:30
   • P5-Clip.wmv: 56:39 - 1:08:44
   • P8-Clip.wmv: 21:40 - 32:00
   • P13-Clip.wmv: 42:40 - 51:20
   • P14-Clip.wmv: 4:58 - 15:37
     
     
3. Use the "Critical Incidents" tab of this analysis spreadsheet to create a content log for each recording, as discussed in class. Pay particular attention to places in the recordings where troubles arise. As you view the segment, feel free to use the following documents to help you understand what is going on (you may also ask your instructor if you are having trouble understanding what is going on):

• The original task instructions given to participants in the P4, P5, and P8 clips (they read these aloud before they perform each task) and the PHASE 2 task instructions given to participants in the P13 clip.
• The SALSA quick reference guide given to participants.
  
  
1. Identify the location of at least one usability problem in each of the clips that have been assigned to you. In the "Usability Problems" tab of your analysis spreadsheet, answer the following questions:
   • What is the usability problem? (Provide a clear, concise description. Don't offer a diagnosis at this time!)
   • Where, in particular, does the usability problem occur? (Provide the specific timer value.)
   • In your estimation, how severe is the problem on the three-point scale of Dumas and Reddish?
   • In your estimation, is the problem likely to affect a broad range of users. Use this three-point scale of Dumas and Reddish.
   • What do you think the cause of the problem is. (Now you can offer a diagnosis!)
   • In order to fix the problem, what specific design recommendation would you make to the ALVIS developers?
     

Deliverables

• Fifteen to 20 minutes prior to the end of class, your instructor will select groups to present the usability problems they found to the class. (The recordings you work with will be available on the computer connected to the projection device.) In particular, you will be asked to (a) describe the problem, (b) show the example footage, (c) diagnose the cause, and (d) suggest a solution.