Assignment 5: Controlled experiment with one participant

In this assignment, you will learn what kind of method is an experimental lab study that can produce data that can be statistically analysed.

We will collectively gather a large quantitative dataset that we will analyse together statistically in the next lecture (L6). For the analysis to succeed, this assignment has a precise protocol which everyone needs to closely follow when carrying out the experiment. The instructions and the protocol are provided here.

Following the example in lecture #5, this study is about the UI design of ribbons. We have two research questions:

  • RQ1: Does a novel kind of "interactive" ribbon design help users to carry out a non-obvious text editing task?
  • RQ2: Does a covert hint, provided to a participant before the actual task, help the participant solve the task?

As a research design, this is so-called 2x2 between-subjects two-way factorial design. Here "2x2" means that the study will compare 2x2 different user interfaces against each other. "Between-subjects" means that each user participates only in creation of one measurement, and therefore the comparisons are made between subjects (i.e., participants). The other option would be "within-subjects" where every participant would carry out all 4 different tasks. "Two-way factorial" means the same as "2x2": that there are two different dimensions (i.e., factors) in the study: we vary both the UI type and the presence of a hint.

This data can be analysed with ANOVA, by using SPSS, R or other statistical software. If the data that you will collectively produce has high enough quality, we will find out the answers to RQ1 and RQ2 are in Lecture 6!

Research instrument

This study is carried out by asking one participant to perform tasks using an instrument that you can find here: https://insightful-interfaces.netlify.com.

The prototype is a simplified version of an ongoing study by Antti Salovaara and Olga Kornilova (Aalto University) and Frederic Vallee-Tourangeau and Wendy Ross (University of Kingston, London). The instrument's development is by Olga Kornilova.

Preparations

Before recruiting a user:

  1. Find out which of the 2x2 conditions your study will be pseudo-randomly assigned to: go to www.sha1-online.com and type your full name to the text field. Look what the last character of the SHA1 hash value is. Use the following rule to find out the study condition:
    1. Last character is 0, 1, 2, or 3: Your condition is "basic with hints".
    2. Last character is 4, 5, 6, 7: Your condition is "basic without hints".
    3. Last character is 8, 9, a, or b: Your condition is "interactive with hints".
    4. Last character is c, d, e, or f: Your condition is "interactive without hints".
  2. Remember what condition was assigned to you!
  3. Visit https://insightful-interfaces.netlify.com and get yourself familiar with the task. On your own, experiment with all the four conditions. 
  4. Make sure that you know how the second task's problem can be solved. THE SOLUTION IS: The solution for the poem problem is that 1) The user selects all the lines, 2) makes it an ordered list, and 3) looks the item number in the last line.
  5. Read these instructions carefully and fully now. Read also the study protocol.
  6. Be prepared to answer the questions that will be asked in the web form below.

Recruit a user:

  1. Read the slides about research ethics from lecture L5.
  2. Find one participant for your study. Your participant must be:
    1. A student in a university or a polytechnic school.
    2. Proficient in understanding English.
    3. Experienced MS Word (or other graphical text editor) user.

The actual study:

  1. Print the experiment protocol linked below or download it in a table. Do not read the protocol from the same computer that you will use in the experiment - switching between screens would  disturb the flow of your experiment!
  2. Carry out the experiment following the instructions in the protocol.
  3. Write down the task completion time (TCTs from the last screen).

Experiment's protocol

The protocol is here. Follow its instructions in the actual experiment.
The informed consent form is here.

How to return this assignment

Similarly to all the other assignments in this course, also is carried out individually.

The assignment is returned by answering to the questions in the web questionnaire (see link below). They will be the following:

  1. What was your study's condition?
  2. What were the task completion times?
  3. Did the participant complete the poem length problem, using the numbered list features, within 3 minutes, without your additional suggestions?
  4. Did you make any errors in the following of the protocol?
  5. Suggest one change to this research design that would improve the validity of this study.
  6. Suggest one change to this research design that would improve the reliability of this study.

Return the assignment by clicking the link below and answering its questions. The deadline is Sunday 13 October at 23:55.

The questionnaire was closed on Monday, 14 October 2019, 2:00 AM. Thanks.