Seminar Advanced Deep Reinforcement Learning 2020

Goal

To have fun exploring Scientific Research. We try to get as close as possible to doing real science as we can in an advanced master course (which some people find extremely satisfying, a.k.a., fun).

Our topic is what is hot in deep reinforcement learning at the moment. Warning: Understanding these papers may be very challenging, which may not be your fault. This is science. The material may be very advanced, the ideas not well crystallized or thought out, or the paper may just not be very well written…

There are survey papers and blogs as introductory/background material, and we study or find cool papers on hot topics to work on, to understand, to implement, to inspire us.

We do so in an interactive, discussion-style setting. This is a research-oriented course. Science is full of questions, together we will work on some answers. My role is more that of a tour guide into the wonderful world of advanced deep reinforcement learning than of someone who has the answers. Other courses provide answers. If you’re looking for questions, you’ve come to the right place.

Overview papers are provided, and links to more state-of-the-art papers. Students choose papers, do presentations, do (partial) implementations, and write a report or paper. There is a possibility, if you are lucky, things go really well, and you have good paper writing skills, that it makes sense to submit the paper to a workshop or conference. In any case, work in this course may be valuable towards your Master’s thesis.

What

  1. Listen to intro lectures
  2. Read/skim all Overview papers
  3. Choose topic curriculum/transfer/meta/model/hierarchical/explainable
  4. Read fully State-of-the-Art Paper (one suggested or your own choice)
  5. Understand, discuss, implement, present, write

When

This course runs in the Fall of 2020. We meet on Mondays at 11:15 from 7 Sept-23 Nov 2020.

Who

The course’s chief cook and bottlewasher is Aske Plaat. Additionally Jan van Rijn, Mike Preuss, Wojtek Kowalczyk, Joost Broekens and Joost Batenburg are interested and may join (to be confirmed, also depending on student interest). The format is modeled on seminar courses at LIACS by Walter Kosters and Hendrik Jan Hoogeboom and others.

Enrolment

This format only works in a small group of motivated people. At most 12 students. Grades for Reinforcement Learning and for Deep Learning/Neural Networks are important for admission. Send email before 1 August to aske.plaat@gmail.com with your grades and an indication of what you are interested in. Oh, ok, 2 August will probably also work. But send a good sounding reason for being late.

Topics

  • Curriculum Learning
  • Transfer Reinforcement Learning
  • Meta Reinforcement Learning
  • Model-based Reinforcement Learning
  • Hierarchical Reinforcement Learning
  • Explainable AI

Overview/Survey/Background Papers/Blogs

To get you started. Start with the BOLD entries. Please read all & make a top 2.

Possible State of the Art papers

But please go look for one yourself.

Many papers also have a blog-version, in which the authors present a more accessible version (informal language, intuitive ideas, more diagrams). Google will find it for you.

Many papers have code on line. These may be the best to start with.

Overview

Work in teams of two.

Choose a paper and really understand it by implementing something; write a report/paper & present on progress.

  • Overview of topics/survey papers
  • Choose topic/state of the art paper
  • Present on progress understanding state of the art paper
  • Implement something related to state of the art paper
  • Present implementation
  • Write report/paper

Lecture Schedule

  1. [Teacher] Intro, Goal, Deep Reinforcement Learning, form teams of two people, start reading survey papers
  2. [Teacher] Review of Topics: Curriculum, Transfer, Meta, Hierarchical, Model-based, Explainable. Choose topics for each team
  3. [Students] Presentation/discussion on first three topics (survey papers).
  4. [Students] Presentation/discussion on second three topics (survey papers). Find state of the art paper to work on
  5. [Students] Feedback by each team how did looking for state of the art paper go
  6. [Students] Presentation/discussion on first three topics implementation progress
  7. [Students] Presentation/discussion on second three topics implementation progress
  8. [Students] Feedback by each team how things are going/issues
  9. [Students] Feedback by each team how things are going/issues
  10. [Students] Final presentation first three topics implementation results
  11. [Students] Final presentation second three topics implementation results
  12. [Students] Write report

Agents/Environments/Benchmarks

More Resources