Topic outline

  • Designing Voice and Auditory Interaction

    Designing Voice and Auditory Interaction

    Advances in voice recognition and digital technologies offer novel forms of speech and auditory interaction for engaging user services and immersive experiences in the home, school, workplace, museums and everyday public spaces. This course introduces methods, practices and challenges of incorporating speech, audio and multimodal interaction for designing voice and auditory services, products and user experiences. Designing Voice and Auditory Interaction provides a basic understanding of speech, conversational design, NLP and interactive speech/audio methods, along with cases studies and tutorials for applied project-based learning, participatory design, and user experience evaluation.     The course will also explore novel research applications for immersive audio environments, physical interaction, and computational creativity.

    Organized as a 5 ECTS seminar, with lectures, visitors, presentations by students and a multi-disciplinary, rapid prototyping project.

    Course Duration: 18.01.2022 - 5.04.2022 (12 weeks)

    Seminar Sessions: Tuesdays 14:00-16:00 

    Project/Tutorial Labs: Thursdays 11:00-12:00

    First meeting on Tuesday 18.1.2022 14:00-16:00. 

    Link to access Online Platforms: Slack, Zoom and Miro will be used during the course (access requires registration to the course).

    Instructors

    • Tom Bäckström (ELEC / SPA)
    • Nitin Sawhney (SCI / CS) 
    • Koray Tahiroglu (ARTS / Media)
    Teaching Assistants
    • Kaisla Kajava (SCI / CS)
    • Mohammad Vali (ELEC / SPA)
    • Camilo Sanchez (ARTS / Media)
    Target Audience

    The course is targeted to students of all schools, in particular ELEC, SCI, ARTS and BUSINESS, at both masters and doctoral levels. 

    Course Structure

    We offer a 5 ECTS seminar course with the title "Designing Voice and Auditory Interaction", where the credits are awarded primarily through "ELEC-E5531 - Speech and Language Processing Seminar". If that course does not fit your studies, we can discuss other routes for awarding credits.

    Learning Goals & Outcomes

    • Seminar presentations of articles and research in topic areas 
    • Open ended participatory research and co-design with potential users 
    • Rapid prototyping of working applications or proof-of-concept designs 
    • User evaluations and feedback to inform design or critical lessons learned 
    • Multi-disciplinary projects in different domains e.g. learning, healthcare, public services, workplace, interactive exhibits etc. 

    Course Schedule (subject to change)

    PART I: FOUNDATIONS 

    Week 1 (18.1.) Introduction, Challenges & Inspirations
      • Why conversational and speech/audio interaction? 
      • Showcasing examples, challenges and opportunities
      • Course Intro: Goals & Expectations
    Week 2 (25.1.) Audio Perception, Phonetics & Speech Production 
      • Sound Perception & Action
      • Phonetics
      • Speech Production
    Week 3 (1.2.) Semantics, Dialogue & Conversational Design 
      • Semantics & Computational Semantics
      • Chatbots & Dialogue Systems
      • Conversational Design
    Week 4 (8.2.) Speech & Auditory Processing for Voice Interaction
      • Speech Recognition & Synthesis
      • Auditory Processing
      • Integrating Speech & Auditory Interaction in Wearable Devices 
    Week 5 (15.2.) Interim Team Presentations
      • Interim Project Proposal Presentations by Teams for feedback & discussion

    PART II: APPLIED RESEARCH & DESIGN 

    Week 6 (22.2.) Designing Multimodal Conversational Systems
      • Presentation by Speechly
      • Dialogue, Conversational Design and Flow 
      • Creating & Testing Voice & Multimodal Content 
    Week 7 (1.3.) Participatory Design of Voice/Audio Interaction
      • Participatory & Co-Design Principles with Users 
      • Review Example Projects & Research Papers 
    Week 8 (8.3.) Examining User Experience & Conducting Evaluation  
      • Using empirical and participatory evaluation methods
      • Designing Voice Assistant Personalities  
      • Designing Voice Interaction with Children: Recent research from user studies
    Week 9 (15.3.) Sound Architectures of Trust, Privacy & Security 
      • Trust, Privacy & Security in Society
      • Distributed vs. centralized speech/audio
      • Building robust & flexible systems in the real-world
    Week 10 (22.3.) Designing Physical & Multimodal Interaction 
      • Physical Interaction
      • Multimodal & Playful Interaction
    Week 11 (29.3.) Computational Creativity & Immersive Environments 
      • Computational Creativity 
      • Designing for Immersive Audio Environments
      • Sound in AR/VR Applications
      • Wearable / Ubiquitous Computing etc.
    Week 12 (5.4.) Final Team Presentations
      • Project Presentations and Feedback (extended 3-hour class 14:00 – 17:00) 

    Team Projects

    • Students may select a thematic focus area – domain (domestic context, learning, healthcare, interactive museum exhibit, public spaces etc.) and demographic group (children, adults, elderly, migrants, special needs group etc.). They must research prior work and related projects in these areas, along with distinct issues and challenges regarding the domain and demographic to propose a potential project. (Weeks 1-2) 
    • Form multi-disciplinary teams (preferably across SCI, ELEC, ARTS, BIZ) and try to find a potential set of end users or an organization to serve as client or context for inquiry, co-design and evaluation. (Weeks 2-3) 
    • Prepare a working project proposal and present interim research for discussion and feedback. (Week 4) 
    • Conduct research, devise proof-of-concept mockups, co-design concepts, do rapid prototyping and working applications and/or devise a user experience and evaluation. (Weeks 5-11) 
    • Share findings and prototypes with a brief write-up (4-5 pages) in the final presentation on 5.4.2022, with feedback and guidance for future work. (Week 12) 
    • OPTIONAL (extra credit 3 points): Teams may future develop the projects and conduct extended user evaluations and trials in field-based contexts to prepare a potential conference publication on their findings (say over 6-8 weeks until end of May if needed). 

    Assessment Criteria

    Students will be evaluated (grades 0 to 5) based on the following criteria:

    • Active participation seminar - interaction, QA (10% of overall grade)
    • Active participation in lab sessions - interaction (15% of overall grade)
    • Team project (75% of overall grade)
      • Interim (25%)
        • Concept presentation (10%)
        • Concept paper (3-5 pages, 15%)
      • Final (50%)
        • Presentation (10%)
        • Prototype (20%)
        • Final paper (8-10 pages, 20%)
    Workload estimation

    5 credits (134 hours) 

    • 12 seminars a 2 h (24 hours participation + 24 hours reflection = 48 h) 
    • 12 Labs (1 hour x 12 = 12 hours)
    • Readings & Reflection (3 hours x 10 = 30 hours)
    • Project work (10 weeks x 5 hours/week = 50 hours) 
    Reading Materials 

    *partial list only, others will be added soon.