Studio Activity/Assignment Option #3:
Heuristic Evaluation & Cognitive Walkthrough

Overview and Learning Objectives

In this In-Class Activity, you will help one of your fellow project groups to refine their project design. In particular, you will use two analytical techniques—heuristic evaluation and cognitive walkthrough—to evaluate another group's "competitor" product (existing software that they reviewed in their project proposal).

The specific learning objectives of this in-class activity are:

• to practice using the heuristic evaluation and cognitive walkthrough to evaluate an existing product;
• to consider the tradeoffs between these two techniques; and
• to use the results of applying these two techniques as a basis for critiquing and improving a design.

Note: At least one group member must have a laptop computer that has installed upon it a software application that is related, or in competition with, the member's group project.

Steps

In this in-class activity, we will form evaluation teams consisting of two to four people from different project groups. Such a mixed project group team will add a bit of objectivity to the analysis exercise: it's better if at least one person in the pair is not familiar with the system being analyzed.

To form a mixed project group team, first find a "buddy" project group. The two "buddy" project groups will form two new evaluation groups, each of which has an equal number of members from each of the two project groups. Each of the two evaluation groups will analyze both group's prototypes. In particular, one evaluation group will perform a heuristic evaluation on prototype A while the other evaluation group performs a heuristic evalution on prototype B. Then, the evaluation groups will swap prototypes. The evaluation group that performed a heuristic evaluation on prototype A will now perform a cognitive walkthrough of prototype B. Likewise, the evaluation group that performed a heuristic evaluation f prototype B will now perform a cognitive walkthrough of prototype A.

1. With your evaluation group, use the heuristic evaluation form to guide you through a heuristic evaluation of software application A. (For more guidance, feel free to make use of this detailed form developed by Xerox.) What potential problems and areas of concern did you encounter? Write them down in the space provided next to each heuristic, along with a brief justification. Note that not all heuristics may be applicable; simply make a note of it if a given heuristic does not apply.

2. With your evaluation group, use the cognitive walkthrough form to guide you through a cognitive walkthrough of software application B. Note that you'll need to choose to focus on three core tasks, and that you'll need to break these three tasks into atomic subtasks for the purposes of the walkthrough. For each subtask, pose the three questions (see the form) and determine whether problems are likely to occur. In the space corresponding to each question, write down any potential problems and areas of concern that were motivated by that question.
   

3. When both evaluation groups are finished with the two analyses, or fifteen minutes before the end of class (whichever comes first), the two project groups should come together to discuss your findings. Consider the following questions:
   • Did you find one or the other technique easier to apply than the other?
   • Did you find problems with one technique that you did not find with the other?
   • Overall, what is your assessment of the two techniques with respect to their relative difficulty and their effectiveness in uncovering problems?
     

Assignment Option #3

Note: As stated in the course syllabus, you are required to hand in at least three assignment options, one from each of the three major sections of the course (Background and Theory, Designing Interactive Systems, Evaluating Interactive Systems). In other words you need to hand in

1. assignment option #1, 2, 3, or 4;
2. assignment option #5, 6, or 7; and
3. assignment option #8.

Assignment Tasks

1. Pick a software application that is either related to, or in direct competition with, your group project. The application may not be one of the applications you evaluated in the studio activity. If you cannot find a suitable competitor application, or if you would like to evaluate a different type of application, then I offer you the following two unusual applications as options: (a) the parking ticket machine at the Student Rec Center; (b) the self-serve check-out kiosk at the Safeway store on Bishop Blvd. in Pullman.

2. Describe the competitor application in a paragraph or two. Include at least one screen shot of the application.

3. Using this heuristic evaluation form as a guide, perform a heuristic evaluation of the application. You may hand in your completed form as part of your write-up of this assignment.

4. Using this cognitive walkthrough form as a guide, perform a cognitive walkthrough of the software application. You may hand in your completed form as part oof your write-up of this assignment.

5. Write a brief "summary of findings" that includes a list of potential problems with the software application, along with the evaluation technique that uncovered the potential problem. For each potential problem, describe a possible solution to the problem.

6. Finally, in a paragraph or two, compare and contrast the two techniques you applied in terms of (a) how easy they were to apply, and (b) their potential to uncover usability problems.

Additional Research Task for Graduate Students

Through the WSU Library E-Journal Access site, access and download the PDF of the following article. (Make sure you're inside the WSU firewall before accessing this document; otherwise, you will not have access to the full PDF of the article.)

Hertzum, M. & Jacobsen, N. (2001). The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods. International Journal of Human-Computer Interaction 13(4), pp.421-443

This article critically reviews the extent to which three usability evaluation methods, including the heuristic evaluation and cognitive walkthrough, are reproducible. Prepare a two page (double-spaced, 1" margins) critical review of the article. Your critical review should not only identify the key points of the material, but also analyze and critique its underlying perspective. The following list of questions provides a useful starting point for your critical review:

• Who are the authors? What is their academic history? From what intellectual perspective are they coming? What are their biases?
• What are the key points of the article?
• Are there any gaps in logic or presentation? How could the presentation or argument of the article be improved?
• How does the article compare, and fit in, to others you've read on this topic (if any)?
• What do you like about the article? What do you not like? Why?

Deliverables and Grading

By the deadline, hand in a hardcopy written report that includes a section that addresses each of the assignment tasks described above. Your report must have a title page with "Assignment Option #3" displayed prominently in bold at the center of the page; your name, the name of this course, and the date should appear in smaller type below the title.

We will use this assessment form to evaluate your work. The critical review required of graduate students will be evaluated according to this assessment form. Be sure you that carefully read these forms ahead of time, so that you can take the assessment framework into account as you write your report.