Assignment 5: Controlled experiment with one participant

Change log:

  • Tuesday 9 Oct: fixed a bug that had made some of the conditions similar to each other
  • Thursday 11 Oct: added an informed consent form and a link to it to this page
  • Thursday 11 Oct: added a welcome page also for the "without hint, baseline UI" condition
  • Thursday 11 Oct: removed a mention to an "information sheet" from the protocol
  • Thursday 11 Oct: added an instruction that the data from the play-test of the experiment should not be submitted with the form.


In this assignment, you will learn what kind of method is an experimental lab study that can produce data that can be statistically analysed.

We will collectively gather a large quantitative dataset that we will analyse together statistically in the next lecture (L6). For the analysis to succeed, this assignment has a precise protocol which everyone needs to follow when carrying out the experiment. The instructions and the protocol are provided here.

Following the example in lecture #5, this study is about the UI design of ribbons. We have two research questions:

  • RQ1: Does a novel kind of "interactive" ribbon design help users to carry out a non-obvious text editing task?
  • RQ2: Does a covert hint, provided to a participant before the actual task, help the participant solve the task?

As a research design, this is so-called 2x2 between-subjects two-way factorial design. It can be analysed with ANOVA, using SPSS or other statistical software, if the data has high enough quality. We will find out what the answers are in Lecture 6!

Research instrument

You must use Chrome in this experiment. You may get error messages with other browsers, because they do not support some of the libraries that the instrument uses.

Click here to access the research instrument: https://users.aalto.fi/~asalovaa/assignment5/

Preparations

  1. With Chrome, visit the web address given above, and get yourself familiar with the tasks. 
  2. Make sure that you know how the problem can be solved. THE SOLUTION IS: The solution for the count problem that we are interested is that 1) The user selects all the addresses, and then 2) makes it an ordered list. The line count can be found by reading the number of the last list item.
  3. Although this study compares 2x2 different situations, your participant will see only one of the four "conditions". The choice between these conditions will be based on pseudo-random assignment based on what a SHA1 hash of your full name is. In the actual experiment you must use your own name as input, but you can inspect the different conditions by entering these dummy names to the instrument:
    1. (with hint, interactive UI): try the instrument with "a" as your name to test this
    2. (with hint, baseline UI): try the instrument with "b" as your name to test this
    3. (without hint, interactive UI): try the instrument with "d" as your name to test this
    4. (without hint,baseline UI): try the instrument with "k" as your name to test this
  4. Read the slides about research ethics from lecture L5.
  5. Read the protocol in its entirety.
  6. "Live-play" the protocol with yourself as the participant. Gather all the requested information and enter it to the assignment's answer sheet (see below), however without pressing the "Submit questionnaire" button. This "pilot test" ensures that you don't miss anything in the actual experiment.
  7. Recruit a participant for your study. Your participant must be:
    1. A student in a university or a polytechnic school.
    2. Proficient in understanding English.
    3. Experienced MS Word (or other graphical text editor) user.
  8. Carry out the experiment following the instructions in the protocol.

Experiment's protocol

The protocol is here. Follow its instructions in the actual experiment.
The informed consent form is here.

Print the protocol and keep it at hand all the time during the study. You can also keep the protocol open in a tablet, if you have one. Do not read the protocol from the computer that you will use in the experiment - that would disturb your experiment!

How to return this assignment

Return the assignment by clicking the link below and answering its questions. The deadline is Sunday 14 October at 23:55.

The questionnaire was closed on Monday, 15 October 2018, 2:00 AM. Thanks.