Advances in voice recognition and digital technologies offer novel forms of speech and auditory interaction for engaging user services and immersive experiences in the home, school, workplace, museums and everyday public spaces. This course introduces methods, practices and challenges of incorporating speech, audio and multimodal interaction for designing voice and auditory services, products and user experiences. Designing Voice and Auditory Interaction provides a basic understanding of speech, conversational design, NLP and interactive speech/audio methods, along with cases studies and tutorials for applied project-based learning, participatory design, and user experience evaluation. The course will also explore novel research applications for immersive audio environments, physical interaction, and computational creativity.
Organized as a 5 ECTS seminar, with lectures, visitors, presentations by students and a multi-disciplinary, rapid prototyping project.
Course Duration: 18.01.2022 - 5.04.2022 (12 weeks)
Seminar Sessions: Tuesdays 14:00-16:00
Project/Tutorial Labs: Thursdays 11:00-12:00
First meeting on Tuesday 18.1.2022 14:00-16:00.
Link to access Online Platforms: Slack, Zoom and Miro will be used during the course (access requires registration to the course).
- Tom Bäckström (ELEC / SPA)
- Nitin Sawhney (SCI / CS)
- Koray Tahiroglu (ARTS / Media)
- Kaisla Kajava (SCI / CS)
- Mohammad Vali (ELEC / SPA)
- Camilo Sanchez (ARTS / Media)
The course is targeted to students of all schools, in particular ELEC, SCI, ARTS and BUSINESS, at both masters and doctoral levels.
We offer a 5 ECTS seminar course with the title "Designing Voice and Auditory Interaction", where the credits are awarded primarily through "ELEC-E5531 - Speech and Language Processing Seminar". If that course does not fit your studies, we can discuss other routes for awarding credits.
Learning Goals & Outcomes
- Seminar presentations of articles and research in topic areas
- Open ended participatory research and co-design with potential users
- Rapid prototyping of working applications or proof-of-concept designs
- User evaluations and feedback to inform design or critical lessons learned
- Multi-disciplinary projects in different domains e.g. learning, healthcare, public services, workplace, interactive exhibits etc.
Course Schedule (subject to change)
PART I: FOUNDATIONSWeek 1 (18.1.) Introduction, Challenges & Inspirations
- Why conversational and speech/audio interaction?
- Showcasing examples, challenges and opportunities
- Course Intro: Goals & Expectations
- Sound Perception & Action
- Speech Production
- Semantics & Computational Semantics
- Chatbots & Dialogue Systems
- Conversational Design
- Speech Recognition & Synthesis
- Auditory Processing
- Integrating Speech & Auditory Interaction in Wearable Devices
- Interim Project Proposal Presentations by Teams for feedback & discussion
PART II: APPLIED RESEARCH & DESIGN
- Presentation by Speechly
- Dialogue, Conversational Design and Flow
- Creating & Testing Voice & Multimodal Content
- Participatory & Co-Design Principles with Users
- Review Example Projects & Research Papers
- Using empirical and participatory evaluation methods
- Designing Voice Assistant Personalities
- Designing Voice Interaction with Children: Recent research from user studies
- Trust, Privacy & Security in Society
- Distributed vs. centralized speech/audio
- Building robust & flexible systems in the real-world
- Physical Interaction
- Multimodal & Playful Interaction
- Computational Creativity
- Designing for Immersive Audio Environments
- Sound in AR/VR Applications
- Wearable / Ubiquitous Computing etc.
- Project Presentations and Feedback (extended 3-hour class 14:00 – 17:00)
- Students may select a thematic focus area – domain (domestic context, learning, healthcare, interactive museum exhibit, public spaces etc.) and demographic group (children, adults, elderly, migrants, special needs group etc.). They must research prior work and related projects in these areas, along with distinct issues and challenges regarding the domain and demographic to propose a potential project. (Weeks 1-2)
- Form multi-disciplinary teams (preferably across SCI, ELEC, ARTS, BIZ) and try to find a potential set of end users or an organization to serve as client or context for inquiry, co-design and evaluation. (Weeks 2-3)
- Prepare a working project proposal and present interim research for discussion and feedback. (Week 4)
- Conduct research, devise proof-of-concept mockups, co-design concepts, do rapid prototyping and working applications and/or devise a user experience and evaluation. (Weeks 5-11)
- Share findings and prototypes with a brief write-up (4-5 pages) in the final presentation on 5.4.2022, with feedback and guidance for future work. (Week 12)
- OPTIONAL (extra credit 3 points): Teams may future develop the projects and conduct extended user evaluations and trials in field-based contexts to prepare a potential conference publication on their findings (say over 6-8 weeks until end of May if needed).
Students will be evaluated (grades 0 to 5) based on the following criteria:
- Active participation seminar - interaction, QA (10% of overall grade)
- Active participation in lab sessions - interaction (15% of overall grade)
- Team project (75% of overall grade)
- Interim (25%)
- Concept presentation (10%)
- Concept paper (3-5 pages, 15%)
- Final (50%)
- Presentation (10%)
- Prototype (20%)
- Final paper (8-10 pages, 20%)
5 credits (134 hours)
- 12 seminars a 2 h (24 hours participation + 24 hours reflection = 48 h)
- 12 Labs (1 hour x 12 = 12 hours)
- Readings & Reflection (3 hours x 10 = 30 hours)
- Project work (10 weeks x 5 hours/week = 50 hours)
*partial list only, others will be added soon.