Assignment 8: User behaviour prediction

Due: Sunday, 10 November 2019, 11:55 PM

Edit log:

5.11.2019: A lot of edits made in order to explicate the purpose of this assignment. Also the instructions have been slightly re-explained, to fill the missing parts that were not discussed in the lecture. 

6.11.2019: Added more information on the search for correct values for a and b.

-------

This assignment familiarises you 

  1. to Fitt's law, which is a simple yet widely used model in HCI, and 
  2. to the advantages of user modeling in "counterfactual" design problems, which are problems where you want to know what users' behaviour will be like if you design a certain kind of a UI. With a good model like Fitt's law, you can do such predictions without needing to recruit users to lab studies. 

These steps are what this assignment is about. You will:

  1. find parameters for the Fitt's law model so that the model will fit the given observed user behaviour. While Fitt's law is universally applicable to pointing tasks (e.g., keyboard typing, point+click tasks, and gesture interrfaces) for all humans, a given human's behaviour can be predicted only after estimating two parameters for that particular person. 
  2. investigate contrafactual ("what if") scenarios. You will change the keyboard layout of in the task, and the estimated parameters and Fitt's law will help you predict how the given users would be able to use the new layout, and thus, whether their performance would improve. This shows the power of user modeling: you can actually use a computer to predict behavioural outcomes for any keyboard layout. This lets you evaluate designs whose research would be impossible in real life because that would require an immense amount of user studies.

Our model is Fitts' law, which was featured in Lecture 8. The use context in this assignment is one finger mobile phone texting. The assumption of one finger makes it easy to model the users' behaviour with a single Fitts' law. Further, we assume that the user does not need to do visual search of buttons, or proof reading of the already typed text.

The source code (fitts-model.py) contains all the code that is required for completing the assignment. The code is documented, and you should read the in-line comments to understand what the functions and function calls do. For simplicity, all the data is hand-crafted for the purpose of this assignment and is not authentic user data.

You can do one of the following: 

  1. Download the Python code from the link above and work with it on your own computer.
  2. Similarly with Assignment 4, run the code in Microsoft Azure notebook by cloning it from this repository to your Azure workspace, editing it there, and running it in the cloud. See the more detailed instructions on cloning and Azure notebooks from Assignment 4.

Step 1. Parameter estimation

Before you can explore the behaviour for any keyboard layout design, you need to estimate Fitt's law's two parameters for a user. In our case we have two users whose parameters you will estimated. The source code contains log-data based text message typing times from two users: data_user_u1 and data_user_u2. These variables contain sentences and how many seconds it took from each user to type those sentences. Your task is to find Fitts' law parameters a and b for these two users. For any two parameters a1 and b1, you can check how well the Fitts' law, equipped with these parameters, corresponds to the user data by calling the function prediction_error() with a1, b1, and the target user data as parameters. 

Your goal is to find such values for a and b, separately for users u1 and u2, that result in as small a prediction error as possible. Prediction error means the difference (in seconds) between the measured typing times (given in data_user_u1 and data_user_u2) and what the Fitt's law model predicts that these times would be. When you find a small prediction error, that means that your model has been calibrated well to describe the user's actual behaviour.

Tip 1: if you don't wish to try different values of a, b one by one, you can use the Python code's function grid_search(), which accepts a list of a and b values, and iterates through all combinations, calculating the prediction error for them. The source code's last row shows how you can call the function and print its results.

Tip 2: the ranges within which a and b may vary can be found from lecture slides, p. 35–36. Added 6 Nov: Looking at Fitt's law's formula helps you understand what a and b mean. Fitt's law tells a movement time (MT) of one pointing action using this formula: MT = a + b log2 (2A / W + 1). This formula is similar to y = a + bx, which is quite familiar from elementary mathematics. Here a is the intercept: the location on y axis when x = 0. The b, in turn is the slope constant. From slides 35-36, you cannot find values for a, but you can find typical values for b. Note that that the tables report them in milliseconds. For estimating the value range for a, notice that this is a value whose metric is seconds. Think what would be a typical range of movement times in phone screen pointing. Can the pointing time be negative? Can one pointing take 100 seconds to carry out? This gives you an idea what range of values there can be for a and b, and you can use the grid_search() function to find the precise estimates.

In your reportpresent the best values for a and b separately for both u1 and u2, and describe how you found out these parameters. In addition, check the lecture slides, p. 35–36, and try to identify which user group u1 and u2 probably represent, given that the Table's "Parameter of interest", "Cycle time of the motor processor" can be matched to parameter a, and "Fitts' law slope constant" can be matched to parameter b.

Note. Fitts' law is a fairly simple model. The reality of typing behaviour is much more extensive than what can be captured with one equation and two parameters. If you are interested, you should follow up on the papers cited in the lecture and considering enrolling to User research course ELEC-E7890 at Aalto the next time it is lectured.

Step 2. Forward modelling

"Forward modelling" refers to using a model for predicting user behaviour of hypothetical ("counter-factual") users. With the parameters now estimated, you will be able to test how well a different keyboard layout would suit for these two users. 

Your counter-factual keyboard layout: For this task, take a look at the specification of a keyboard layout in the Python code (see the keyboard variable in the source code). Using the existing layout as your example, create a new keyboard layout that follows the Dvorak simplified keyboard layout. You only need to replicate the lowercase letters a–z and the spacebar.

Your task is to predict the typing time for the sentence "how fast can you come to my place" for both users u1 and u2, with both Qwerty and Dvorak layouts. Use the user-specific parameters for the Fitts' a and b that you uncovered in Task 1. Use the call fitts_typing_time that in the Python code is used within print function to obtain results. 

In your report, present a table like this with the typing times in seconds entered in the empty cells:

          User 1   User 2
Dvorak
Qwerty

Note. In this assignment, we assume that the users are experts with both Qwerty and Dvorak layouts. For more realistic predictions, you would need some sort of a visual search model and keyboard layout learning model. See the lecture slides for an introduction to them.

Step 3. Reflecting the value of modelling

Provide an example of a situation (different than the one in this assignment) where a model of a user and a possibility to use it to predict user's behaviour can be beneficial. The model can, for instance, help in improving system's usability or providing useful system features. Your example can be also from some other situation than text typing.

Report submission

Save your report in PDF and submit it at the latest on Sunday 11 November at 23:55.