ELEC-E8125 - Reinforcement learning D, Lecture, 13.9.2021-8.12.2021

Topic outline

• Lectures
  Concept

  Lectures will be places of discussion where the current topic is summarized by the lecturer and discussed among all present. The students are expected to prepare by reading given material in advance prior to each lecture.
  

  Arrangements
  

  Course lectures will be given during first and second periods on Tuesdays 14:15-16:00. All lectures will be given over Zoom at link https://aalto.zoom.us/j/63645678644. Please download and install Zoom before the first lecture to attend the course. Lecture recordings will likely be available afterwards, but this cannot be guaranteed due to potential technical issues. The lectures are interactive in nature so that participation is encouraged.

  Course lectures will be given by Ville Kyrki (first part) and Joni Pajarinen (second part).
  

  Schedule and Readings
  

  For each lecture starting from the third one, there will be reading materials that the students should study before attending the lecture.

  Course arrangements, Overview, Tue 14.9., no readings
  

  Markov decision processes, Tue 21.9., Sutton & Barto, chapters 2-2.3, 2.5-2.6, 3-3.8

  RL in discrete domains (value-based RL), Tue 28.9., Sutton&Barto Ch. 5-5.4, 5.6, 6-6.5

  Function approximation, Tue 5.10., Sutton&Barto Ch. 9-9.3, 10-10.1

  Policy gradient, Tue 12.10.,  Sutton&Barto, Ch. 13-13.3
  

  Actor-critic, Tue 19.10., Sutton & Barto, Ch. 13.5, 13.7
  

  Towards model-based reinforcement learning: optimal control, Tue 26.10. Platt: Introduction to Linear Quadratic Regulation
  

  Model-based reinforcement learning, Tue 2.11., Sutton & Barto, Ch. 8-8.2

  Guest lectures Tue 9.11.: Safety and constraints (Gökhan Alcal), Entropy Regularization in Reinforcement Learning (Riad Akrour)

  
  

  Partially observable MDPs, Tue 16.11., Anthony Cassandra, POMDP tutorial, http://www.pomdp.org/tutorial/, steps from "Brief Introduction to MDPs" until " General Form of a POMDP solution".
  

  Large POMDPs, Tue 23.11.

  
  

  Project show!, to be confirmed

  • Select activity Lecture slides

    Completion

    Students must

    

    Mark as done

    Lecture slides

  • Select activity Course arrangements

    

    Course arrangements File PDF
  • Select activity Lecture 1: Overview

    

    Lecture 1: Overview File PDF
  • Select activity Lecture 2: Markov decision processes

    

    Lecture 2: Markov decision processes File PDF
  • Select activity Lecture 3: Reinforcement learning

    

    Lecture 3: Reinforcement learning File PDF
  • Select activity Lecture 3 recording (Aalto login only)

    

    Lecture 3 recording (Aalto login only) URL
  • Select activity Lecture 4: Function approximation

    

    Lecture 4: Function approximation File PDF
  • Select activity Lecture 4 recording (Aalto login only)

    

    Lecture 4 recording (Aalto login only) URL
  • Select activity Lecture 5: Policy gradient

    

    Lecture 5: Policy gradient File PDF
  • Select activity Lecture 5 recording (Aalto login only)

    

    Lecture 5 recording (Aalto login only) URL
  • Select activity Lecture 6: Actor-critic methods

    

    Lecture 6: Actor-critic methods File PDF
  • Select activity Lecture 6 recording (Aalto login only)

    

    Lecture 6 recording (Aalto login only) URL
  • Select activity Lecture 7: Optimal control

    

    Lecture 7: Optimal control File PDF
  • Select activity Lecture 7 recording (Aalto login only)

    

    Lecture 7 recording (Aalto login only) URL
  • Select activity Lecture 8: Model-based reinforcement learning

    

    Lecture 8: Model-based reinforcement learning File PDF
  • Select activity Lecture 8 recording (Aalto login only)

    

    Lecture 8 recording (Aalto login only) URL
  • Select activity Guest lecture of Gökhan Alcan

    

    Guest lecture of Gökhan Alcan File PDF
  • Select activity Guest lecture of Gökhan Alcan recording (Aalto login only)

    

    Guest lecture of Gökhan Alcan recording (Aalto login only) URL
  • Select activity Guest Lecture on Entropy Regularization in Reinforcement Learning (Riad Akrour)

    

    Guest Lecture on Entropy Regularization in Reinforcement Learning (Riad Akrour) File PDF
  • Select activity Guest Lecture on Entropy Regularization in Reinforcement Learning recording (Aalto login only)

    

    Guest Lecture on Entropy Regularization in Reinforcement Learning recording (Aalto login only) URL
  • Select activity Lecture 9: Partially observable Markov decision processes (POMDPs)

    

    Lecture 9: Partially observable Markov decision processes (POMDPs) File PDF
  • Select activity Lecture 9 recording (Aalto login only)

    

    Lecture 9 recording (Aalto login only) URL
  • Select activity Lecture 10: Larger POMDPs

    

    Lecture 10: Larger POMDPs File PDF
  • Select activity Lecture 10 recording (Aalto login only)

    

    Lecture 10 recording (Aalto login only) URL
  • Select activity Readings and other materials

    Completion

    Students must

    

    Mark as done

    Readings and other materials
    

  • Select activity Sutton & Barto, Reinforcement Learning: An introduction, 2nd ed.

    

    Sutton & Barto, Reinforcement Learning: An introduction, 2nd ed. URL

    Completion

    Students must

    

    Mark as done
  • Select activity Extra slides: Planning in discrete space

    

    Extra slides: Planning in discrete space File PDF

    Completion

    Students must

    

    Mark as done
  • Select activity Platt, Introduction to Linear Quadratic Regulation

    

    Platt, Introduction to Linear Quadratic Regulation URL

    Completion

    Students must

    

    Mark as done
  • Select activity Reinforcement learning course at UCL

    

    Reinforcement learning course at UCL URL
  • Select activity Deep reinforcement learning course at UC Berkeley

    

    Deep reinforcement learning course at UC Berkeley URL
  • Select activity LaValle, Planning Algorithms

    

    LaValle, Planning Algorithms URL

    Completion

    Students must

    

    Mark as done
  • Select activity Lecture slides from 2020

    Completion

    Students must

    

    Mark as done

    Lecture slides from 2020
    

  • Select activity Course arrangements

    

    Course arrangements File PDF
  • Select activity Lecture 1: Overview

    

    Lecture 1: Overview File PDF
  • Select activity Lecture 1 recording (requires Aalto account)

    

    Lecture 1 recording (requires Aalto account) URL
  • Select activity Lecture 2: Markov Decision Processes

    

    Lecture 2: Markov Decision Processes File PDF
  • Select activity Lecture 2 recording (Aalto login only)

    

    Lecture 2 recording (Aalto login only) URL
  • Select activity Lecture 3: Reinforcement learning in discrete domains

    

    Lecture 3: Reinforcement learning in discrete domains File PDF
  • Select activity Lecture 3 recording (Aalto login only)

    

    Lecture 3 recording (Aalto login only) URL
  • Select activity Lecture 4: Function approximation

    

    Lecture 4: Function approximation File PDF
  • Select activity Lecture 5: Policy gradient

    

    Lecture 5: Policy gradient File PDF
  • Select activity Lecture 5 recording (Aalto login only)

    

    Lecture 5 recording (Aalto login only) URL
  • Select activity Lecture 6: Actor-critic methods

    

    Lecture 6: Actor-critic methods File PDF
  • Select activity Lecture 6 recording (Aalto login only)

    

    Lecture 6 recording (Aalto login only) URL
  • Select activity Lecture 7: Optimal control

    

    Lecture 7: Optimal control File PDF
  • Select activity Lecture 7 recording (Aalto login only)

    

    Lecture 7 recording (Aalto login only) URL
  • Select activity Lecture 8: Model-based RL

    

    Lecture 8: Model-based RL File PDF
  • Select activity Lecture 8 recording, only partial available (Aalto login only)

    

    Lecture 8 recording, only partial available (Aalto login only) URL
  • Select activity Guest lecture: Safety and optimal control (Gökhan Alcan)

    

    Guest lecture: Safety and optimal control (Gökhan Alcan) File PDF
  • Select activity Guest lecture recording (Aalto login only)

    

    Guest lecture recording (Aalto login only) URL
  • Select activity Guest lecture notes

    

    Guest lecture notes File PDF
  • Select activity Lecture 9: Partially Observable Markov Decision Processes

    

    Lecture 9: Partially Observable Markov Decision Processes File PDF
  • Select activity Lecture 9 recording (Aalto login only)

    

    Lecture 9 recording (Aalto login only) URL
  • Select activity Lecture 10: Large POMDPs

    

    Lecture 10: Large POMDPs File PDF
  • Select activity Lecture 10 recording (Aalto login only)

    

    Lecture 10 recording (Aalto login only) URL
  • Select activity Extra links

    Extra links
    

  • Select activity Model-based reinforcement learning tutorial (ICML 2020)

    

    Model-based reinforcement learning tutorial (ICML 2020) URL
  • Select activity Bradberry, "Introduction to Monte Carlo Tree Search"

    

    Bradberry, "Introduction to Monte Carlo Tree Search" URL
  • Select activity Deep Reinforcement Learning: Pong from Pixels

    

    Deep Reinforcement Learning: Pong from Pixels URL
  • Select activity AlphaGo: using machine learning to master the ancient game of Go

    

    AlphaGo: using machine learning to master the ancient game of Go URL
  • Select activity Andrew Ng, "Nuts and Bolts of Applying Deep Learning" (video)

    

    Andrew Ng, "Nuts and Bolts of Applying Deep Learning" (video) URL
  • Select activity Juliani, Simple Reinforcement Learning with Tensorflow

    

    Juliani, Simple Reinforcement Learning with Tensorflow URL
  • Select activity Slides from 2019

    Slides from 2019
    

  • Select activity Guest lecture: Deep reinforcement learning in robotics

    

    Guest lecture: Deep reinforcement learning in robotics File PDF