Översikt

  • Course home page

    Welcome to learn the methods for evaluating user interfaces!

    Ongoing evaluation of a product's usability and user experience is a key to the design of any successful service. This hands-on course teaches two main approaches for evaluation: expert-based ("heuristic") evaluations without users, and empirical usability evaluation with users.

    After this course, you will be able to apply these methods to product design processes on different levels of rigour, adapting the methods to the time and resources that are available. You are also able to understand the principles behind the different usability criteria and why even a small number of test users can suffice for reliable conclusions on usability.

    You will have a chance to apply the methods to real products from several different companies, including Telia, Elisa, Helsinki Region Transport (HSL), Finnish Design Shop, Children's Hospital, Ajeco and Trimble. This list is still subject to change.

    The course consists of the following phases:
    1. Introductory lectures and reading material, ending with an exam
    2. Evaluation planning in cooperation with the customer
    3. Expert-based evaluation and a presentation of results in the course
    4. Empirical evaluation with users and a presentation of results in the course
    5. Final report creation, presentation in the course, and delivery to the customer
    Please have a look at the Schedule for more details on the phases and the Syllabus for grading, eligibility to the course, and work load calculation.

    Welcome!

    Antti Salovaara
    Aqdas Malik
    Nina Karisalmi
    Marko Nieminen

  • Schedule + Lecture slides

    Date Type Content
    Tue 26.2.2019 at 12-14 (A136) Lecture Introduction
    Practical issues + grading principles
    Previews about customers' products that will be evaluated
    Fri 1.3.2019 at 10-12 (TUAS TU5) Lecture Basics on usability
    Expert evaluation methods
    Tue 5.3.2019 at 12-14 (A136) Lecture Empirical evaluation methods
    Research designs & research with small sample sizes
    Fri 8.3.2019 at 10-12 (TU5) Lecture Empirical methods and validity threats - Guest lecture by Jussi Jokinen (Aalto ELEC, User Interfaces group)
    Tue 12.3.2019 at 12-14 (T3) Exam Exam on introductory readings
    Fri 15.3.2019 at 10-12 (TU5) Lecture + Group work Lecture on correct answers to the exam
    Planning for customer visit + overall project work
    Tue 19.3.2019 at 12-14 (A136) Lecture Usability evaluation in a corporate context - Guest lecture by Esko Kurvinen (Service Design Lead, Elisa)
    20-28.3.2019 Group work Meetings with customers
    Research plan creation
    Optional tutor meeting Tue 26.3.2019 at 12-14 (A136)
    Fri 29.3.2019 at 10-12 (TU5) Presentations Project plans
    Deadline for project plan submission Fri 23:55.
    30.3.-4.4.2019 Group work Conducting expert evaluations
    Optional tutor meeting Tue 2.4.2019 at 12-14 (T3)
    Fri 5.4.2019 at 10-12 (TU5) Presentations Results from expert evaluations
    Deadline for expert evaluation submission Fri 23:55.
    6.4.-29.4.2019 Group work Conducting empirical evaluations
    Optional tutor meeting Tue 16.4.2019 at 10-12 (T3) + as needed
    Tue 30.4.2019 at 12-14 (A136) Presentations Results from empirical evaluations
    Deadline for empirical evaluation report submission Fri 23:55.
    Tue 7.5.2019 at 12-14 (A136) Presentations Final presentations (4 groups) ** Cancelled **
    Tue 14.5.2019 at 12:15-15 (T6) Presentations Final presentations (all groups à 15 mins)
    Tue 21.5.2019 Final report deadline Final report shipping to the customer (with a tutor in Cc)
    Thu 23.5.2019  Learning diary deadline  Submission of the personal learning diary

  • Reading material

    The pre-exam will cover the lecture contents and the following texts (not including item listed in  Optional sources).

    NEW: The required reading materials can now be downloaded from the "For Aalto users" section of these course pages.

    Expert methods

    Heuristic evaluations:

    • Lewis, C. & Rieman, J. (1994) Task-Centered User Interface Design, Heuristic Analysis. HTML (if the website is inaccessible, download the entire chapter from here)

    Walkthroughs:

    • Lewis, C. H. and Rieman, J. (1994) Task-Centered User Interface Design, Chapter 4.1: Cognitive Walkthroughs HTML (if the website is inaccessible, download the entire chapter from here)
    • Sears, A. (1997) Heuristic Walkthroughs: Finding the Problems Without the Noise. International Journal of Human-Computer Interaction, 9(3), pp. 213-234. DOI: 10.1207/s15327590ijhc0903_2
    Optional sources:

    • Molich, R. and Nielsen, J. (1990) Improving a human-computer dialogue. Communications of the ACM, 33, pp. 338-348. DOI: 10.1145/77481.77486
    • Nielsen, J. (1993) Usability Engineering. Elsevier.
    • Riihiaho, S. (2002) The pluralistic usability walk-through method. Ergonomics in Design: The Quarterly of Human Factors Applications. Vol. 10, No. 3, pp. 23-27. DOI: 10.1177/106480460201000306
    • Bias, R. (1994): The pluralistic usability walkthrough: Coordinated empathies. In Nielsen, J. & Mack R.L. (Eds.) Usability inspection methods. John Wiley & Sons.
    • Kontio, J., Lehtola, L. & Bragge J. (2004). Using the focus group method in software engineering: obtaining practitioner and user experiences. Proceedings of the 2004 International Symposium on Empirical Software Engineering (ISESE'04). IEEE. DOI: 10.1109/ISESE.2004.1334914

    Empirical methods

    Experimental research methodology:

    • Hornbæk, K. (2013): Some Whys and Hows of Experiments in Human-Computer Interaction. Foundations and Trends in Human–Computer Interaction. Vol. 5, No. 4 (2011) 299–373. DOI: 10.1561/1100000043
    • Ritter, F. E., Baxter, G. D. and Churchill, E. F. (2014). Methodology III: Empirical Evaluation. Chapter 13 (pp. 353-400) in Ritter et al., Foundations for Designing User-Centered Systems: What System Designers Need to Know about People. The entire book can be downloaded in PDF through Aalto Library.

    Usability testing:

    • Hornbæk, K., 2006. Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies 64(2), pp.79-102. DOI: 10.1016/j.ijhcs.2005.06.002
    • Kjeldskov, J., Skov, M.B. & Stage, J., 2004. Instant data analysis: conducting usability evaluations in a day. In Proceedings of the third Nordic conference on Human-computer interaction. ACM, pp. 233-240. DOI: 10.1145/1028014.1028050

    Optional material:

    • Riihiaho, S., 2015. Experiences with usability testing: Effects of thinking aloud and moderator presence, Chapter 3.1 (pp. 28-43). PhD thesis. Aalto University. http://urn.fi/URN:ISBN:978-952-60-6227-3.
    • Gray, W. D. & Salzman, M . C. (1998) Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods, Human–Computer Interaction 13(3), 203-261, DOI: 10.1207/s15327051hci1303_2
    • Nielsen Norman Group: Turn User Goals into Task Scenarios for Usability Testing. HTML

    Methodology reviews and special considerations

    Increasing believability of prototype-based studies:

    • Salovaara, A., Oulasvirta, A. & Jacucci, G. (2017). Evaluation of prototypes and the problem of possible futures. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2017), pp. 2064-2077. DOI: 10.1145/3025453.3025658

    Optional material:

    • Sauro, J., Lewis, J. R. (2012) Quantifying the user experience - Practical statistics for user research. Chapter 2: Quantifying User Research. Elsevier.
    • Maguire, M. (2001): Methods to support human-centred design. International Journal of Human-Computer Studies 55(4), 587-634.DOI: 10.1006/ijhc.2001.0503
    • Tyllinen, M., Kaipio, J., Lääveri, T. and Nieminen, M., 2017. End-Users' Voice in EHR Selection: Development of a Usability Questionnaire for Demonstrations in Procurement (DPUQ). Studies in health technology and informatics. 234, pp.346-351. http://ebooks.iospress.nl/publication/46190 DOI: 10.3233/978-1-61499-742-9-346

  • Groups' reports

  • Learning diary

    ** NEW: DEADLINE MOVED TO SUNDAY NIGHT - NEW DL IS 26 MAY 23:55 **

    The purpose of the diary is to provide you with an additional opportunity to learn about usability evaluations and research methodology. The learning diary is less connected to the schedule and hurdles of your project, and this allows you to think about methodological problems in more detail.

    The diary is written as an individual exercise, but discussing its contents with others is permitted (and even encouraged!). 

    You must organize your diary into the sections and subsections that are given below.

    Each of the 5 sections is evaluated with 0–4 points, giving you a total of 20 points from the diary. The recommended maximum length of the diary is 600 words / section = 3000 words / entire diary. 600 words is approximately 2 pages of text with 1.5 line spacing.

    High points will be awarded to those diaries that:

    1. Relate their thoughts to literature (academic and/or practitioner-oriented).
    2. Are “confessional”, i.e., reflect on problems that were encountered within the group project, analyse them, and identify hindsight solutions on how the challenge could have been tackled in a better way.

    Section 1. A usability evaluation challenge

    Choose one challenge or problem that your diary will be focused on and which relates to the evaluation project that you are doing during this spring. You can choose challenge suggested by e.g., literature that you read for the pre-exa, or you can adapt such a challenge for your needs, or you can pick your own challenge. Examples of challenges are (these are sampled from Hornbaek 2006, p. 97–98, see the Conclusion section):

    • How the quality of interaction can and should be evaluated?
    • How could the outcome of interaction be better evaluated?
    • How could usability evaluations better address long-term usability instead of just first-time use of a system?
    • How could objective and subjective measures be better triangulated or used together in usability evaluations?
    • How could satisfaction be evaluated other ways in addition to post-study interviews and questionnaires?

    Remember that these are just examples. Your challenge may also focus on other matters, such as:

    • How to evaluate products whose user group differs a lot of your personal life experience?
    • How a particular aspect of usability (e.g., safety) could be better evaluated?
    • How realism (ecological validity) could be improved in a particular type of evaluation that your project exemplified?
    • etc.

    Organise this section into the following subsections:

    1.1 The  research challenge (express the challenge in a question format)

    1.2 How the challenge related to the project

    1.3 Why this challenge was important

    1.4 Background material (e.g., literature) related to the challenge

    Section 2. Research planning

    Related to the challenge that you chose in Section1:

    2.1 Methodological obstacles and opportunities (related to the challenge at the planning phase)

    2.2 Methodological choices (what methods did you plan to use in the project)

    2.3 Methods vs the challenge (how do the methods relate to the challenge: e.g.,, did they take the challenge  into account)

    2.4 Hindsight (what would have been a better plan that would have better tackled the challenge?)

    Section 3.Expert evaluation

    Related to the challenge that you chose in Section1:

    3.1 Methods (what methods did you actually apply in your project)

    3.2 Validity problems (what problems did your methods have, related to the challenge)

    3.3 Hindsight (what changes would you do to your methods if you would carry out the evaluation again)

    Section 4. Empirical evaluation

    Related to the challenge that you chose in Section1:

    4.1 Methods (what methods did you actually apply in your project)

    4.2 Validity problems (what problems did your methods have, related to the challenge)

    4.3 Hindsight (what changes would you do to your methods if you would carry out the evaluation again)

    Section 5. Final report

    Related to the challenge that you chose in Section1:

    5.1 Limitations (overall, how did the challenge limit the generalizability of your results)

    5.2 Improvements (your suggestions on how to improve usability evaluation methods so that they better take into account the challenge that you had chosen)

  • För Aalto

    Tillgänglig om: Fältet User account contains (use: aalto.fi) innehåller aalto.fi

  • Exam questions


    Following is the list of questions that covers all the lectures delivered in the course. Please prepare all the listed questions thoroughly. In exams, you will get only 3 questions from the following list to answer. Good Luck!

    • How does methodological triangulation improve validity of a usability study?
    • What are the strengths and weaknesses of the heuristic evaluation method?
    • What way would you decide which set of usability heuristics you would use in an evaluation of a product’s usability?
    • Describe three different methods that can be used to measure effectiveness.
    • Describe three different methods that can be used to measure efficiency.
    • Describe three different methods that can be used to measure satisfaction.
    • What are the pros and cons of a narrow research question?
    • Describe three ways with which you can help the usability study participant act naturally in a lab setting.
    • Let’s imagine that you will be evaluating the standard mobile-based SMS editor. Invent three different research questions (one open-ended, one close-ended, and one yes/no question) that a heuristic evaluation could address. For each research question, explain the reason why it would be considered a good question.
    • Let’s image that you will be evaluating the standard mobile-based SMS editor. Invent three different research questions (one open-ended, one close-ended, and one yes/no question) that an empirical usability evaluation could address. For each research question, explain the reason why it would be considered a good question.
    • Present three different biases / validity threats in usability tests with users and how they can be overcome.
    • Why are scenarios so important in both walkthroughs and empirical usability evaluations?
    • Develop a brief scenario that can presented to the users to test the usability of a social map application.
    • How can be usability problems related to the gulf of evaluation be studied?
    • How can be usability problems related to the gulf of execution be studied?
    • It is better to carry out the expert evaluation first and only after that the empirical evaluation. Why?
    • Using the concepts of reliability and/or validity, explain why it is a good idea that a heuristic evaluation is carried out by 3–5 evaluators.
    • What is think aloud protocol? How is it applied within the context of usability testing?
    • Consider a heart rate monitoring machine installed in a hospital. Briefly describe three critical aspects of usability from the perspective of nursing staff and patients. Explain why you identified these aspects in particular.
    • List down any three Nielsen's usability heuristics, and briefly explain each of them with an example.
    • What would be your sampling (recruitment) strategy if you intend to test a newly designed ATM?

  • How to claim back costs

    You can, for example, get your money back for these kinds of expenses:

    • Participation compensations. For simplicity, you can only offer things (not plain money or money-equivalent gift cards, to "S" or "K" chain stores or Stockmann, for example) and max 20 eur/person. Movie tickets are a safe choice. Your customer company may have different regulations, so it is a good idea to ask them.
    • Travel costs
    • Material costs

    If you claim back movie tickets or other compensations    , ask participant details and a signature from each participant, following these regulations (available in Finnish only). You can use this document to gather the necessary information.

    In all these cases, we’d like to ask you first pay the purchases from your own wallet. You can then claim your money back from us. To claim your money back, follow these steps:

    1. Save all the receipts so that you can prove that the cost is real.
    2. Scan all the receipts
    3. Prepare a document that details what these costs were: what their purpose was
    4. Fill in a personal details form. You can download the form from this page.
    5. Send all the material (scanned receipts, the cost explanation document, the personal details form) to Antti when you know that there will be no further costs

    Do these steps individually for each person who made purchases.

    I can then send this information to Travel Services, which is the unit that will pay you back.