In-Class Activity #8:
User Modeling

Overview and Learning Objectives

In this In-Class Activity, you will practice applying the user modeling techniques described in class to your project group's evolving prototype.

The specific learning objectives of this in-class activity are:

• to practice applying the GOMS cognitive modeling technique;
• to practice applying the Keystroke Level Model; and
• to consider the tradeoffs between these two techniques.

Notes and Resources

• Please bring your project group's low fidelity prototype to Thursday's class. You will be evaluating it.
• Please work on this activity with your project group
• Please consult the lecture slides, pp. 449-452 of the text, and the KLM supplement, for guidance on how to construct GOMS and KLM models

Steps

For last week's prototype walkthroughs, your project group identified five core tasks, and developed a low fidelity prototype with which a user could accomplish those tasks. In this exercise, you will apply the two user modeling techniques discussed in Tuesday's lecture to those five tasks.

1. Perform a GOMS analysis of the five core tasks. Be sure to identify all four components of the model for each task: Goals, Operators, Methods, and Selection Rules. For practice, I suggest using the format of the text editing example in the lecture slides.
2. Perform a Keystroke Level Model analysis of the five core tasks. Consider each alternative action sequence for each task. Be sure clearly document your assumptions. Come up with a concrete time prediction for each alternative action sequence of each task.
3. Fifteen minutes prior to the end of class, the instructor will lead a discussion in which you will be invited to share your experiences and describe the tradeoffs between the two techniques.
   

Assignment #8

This assignment gives you the opportunity to obtain further practice in

• observing humans using computers;
• applying task analysis, in which a human procedure is decomposed into atomic steps; and
• applying analytical methods (keystroke anlaysis and GOMS) to obtain predictions of human performance.

Note: As stated in the course syllabus, you are required to hand in at least three assignment options, one from each of the three major sections of the course (Background and Theory, Designing Interactive Systems, Evaluating Interactive Systems). In other words you need to hand in

1. assignment option #1, 2, 3, or 4;
2. assignment option #5, 6, or 7; and
3. assignment option #8 (your only option!)

Assignment Tasks

Consider the following simple word processing task in Microsoft Word:

1. Type in "Is this a dagger that I see before me, the handle toward my hand?"
2. Replace "dagger" with "squirrel".
3. Replace "handle" with "bushy tail".
4. Italicize "hand".
5. On a new line, type "Come, let me clutch thee."
6. On a new line, type "I have thee not yet I see thee still."
7. Replace all instances of "thee" with "you".
8. Replace "me" with "my very eyes".
9. Underline "squirrel".
10. Save the document as "tail.doc".

Launch Microsoft Word and try the task yourself. Consider the following questions:

• In how many different ways might this task be done?
• Are some ways more efficient than others?
• How much more efficient?
• In general, how long can we expect Microsoft Word users to perform each individual step of this task?

Note that, with the exception of the first one, all of these questions are essentially empirical. While we may be able to make some reasonable guesses, we won't be able to know the answers for sure until we actually run an empirical study. In this assignment, you'll do just that, and you'll use your results to develop a couple of predictive models of human performance on these tasks.

There are four key steps in this assignment: (a) performing a task analysis; (b) running a mini empirical study in which you observe three of your friends as they work through the above word processing tasks with Microsoft Word; (c) analyzing your data; and (d) developing a predictive model of human task performance based on your data.

Step 1: Perform a task analysis to build a predictive model of human performance

Perform the task on your own, and write down the human procedure you used to perform the tasks. Now use your procedure as a basis for constructing two alternative predictive models of the task: (a) GOMS, and (b) keystroke. See the lecture slides, pp. 449-452 of the text, and the KLM supplement, for guidance on how to construct these models. According to the keystroke model, how long will it take someone to perform the entire task, and each subtask?

Step 2: Run mini empirical study

Now it's time to gather some empirical data to compare against your predictions. Recruit at least three friends or acquaintances to participate in a mini empirical study. (Note: You may not recruit students enrolled in this course!) Tell them that you need about 15 minutes of their time, and that they will be helping you with a class assignment by practicing a simple task with Microsoft Word. Note that all of your participants must be proficient with Microsoft Word. If you choose people who have never used Word before, your results are likely to be a poor fit to your models.

To get you in the habit of running a proper empirical study, I would like you to have all of your participants read and sign this informed consent form (you will need to modify it slightly so that it applies to your study) prior to their participation.

After your participants sign the informed consent form, please have them follow along as you read the following set of instructions. Note that you will have to modify these instructions with more specific task instructions based on the human procedure you come up with in Step 1 above.

You are required to record participants as they practice the task. Here, you have at least two options:

1. Use your own video camera to capture their computer screen over their shoulder. If you choose this option, bear in mind that LCD flat panel displays will record much better than traditional monitors. (They do not cause the flicker that you see on recordings of traditional monitors.) Also bear in mind that your recording must be good enough that you can discern the individual steps that participants are doing, because you will ultimately need to time each of these steps.
2. Use Camtasia Studio. Surf to http://www.techsmith.com/products/studio/default.asp and download a free trial version of Camtasia Studio software. This software allows you to capture a high fidelity AVI movie of participants' computer screen, including their voices. Read the excellent user documentation that comes with Camtasio Studio for specific instructions on how to use the software.

It is imperative that you record at least one trial for each participant in which the participant performs the task as quickly as possible and without errors. That way, your empirical data stands the best chance of matching your predictions. (Remember, the KLM predicts practiced error-free behavior!)

Step 3: Compile and analyze data

After you have run your participants through the study, use your recordings to compile the following data on each participant:

• Time to complete the whole task.
• Time to complete each of the 10 subtasks.
• Any other notable observations you can make. For example, any difficulties that arose, any deviation from the prescribed method, or any errors that were made.

To help you organize and interpret your data, compile it into a table like the following:

In the "Comments" column, note any discrepancies in the ways in which participants completed the subtask. In addition, be sure to compute the average subtask (in the "Average" column) and total task completion times across all participants (in the "Total time" row).

Step 4: Compare your data to your predictions

The data you gathered in the previous step can help you to refine the predictive model you completed in Step 1. Compare your data to the times predicted by your models. How well did your model do? Quantify this by computing the percent error for your model:

percent error = ( abs( observed time - predicted time ) / predicted time ) * 100

(A percent error of 20 percent or less is considered good.) Where did your model go wrong? What parameters would you need to adjust to bring your model into accordance with your data? Discuss.

Additional Research Task for Graduate Students

Read the ACM SIGCHI 2004 best-paper article "Model-based evaluation of cell phone menu interaction " by R St. Amant, T. Horton, and F. Ritter. It is available on-line at http://doi.acm.org/10.1145/985692.985736. (Make sure you're inside the WSU firewall before accessing this document; otherwise, you will not have access to the full PDF of the article.)

This article tests a GOMS ACT-R models for predicting human performance in common cell phone menu navigation tasks. Prepare a two page (double-spaced, 1" margins) critical review of the article. Your critical review should not only identify the key points of the material, but also analyze and critique its underlying perspective. The following list of questions provides a useful starting point for your critical review:

• Who are the authors? What is their academic history? From what intellectual perspective are they coming? What are their biases?
• What are the key points of the article?
• Are there any gaps in logic or presentation? How could the presentation or argument of the article be improved?
• How does the article compare, and fit in, to others you've read on this topic (if any)?
• What do you like about the article? What do you not like? Why?

Deliverables and Grading

By the deadline, hand in a written report that includes the following sections:

• an abstract that summarizes what you did, and reports your key results (abstracts are really important but difficult to write; do your best);

• a section that derives your GOMS and keystroke models for the task;

• a section that briefly describes your mini empirical study (the procedure participants followed and how you recorded participants);

• a section that presents your data. You should include the above table in this section.

• a section that compares your data to your original predictions, and discusses the questions listed in Task 4: namely,
  • Where did your model go wrong (if it did)?
  • What parameters would you need to adjust to bring your model into accordance with your data?
  • If you could do it again, how would you have done things differently?

Hand in a hardcopy of your document to your instructor at the beginning of class on the due date.

We will use this assessment form to evaluate your work. The critical review required of graduate students will be evaluated according to this assessment form. Be sure you that carefully read these forms ahead of time, so that you can take the assessment framework into account as you write your report.