Topic outline

  • General

    About this course

    This course examines dynamic optimization models; i.e. models that are defined over continuous time, multiple stages, etc. They capture many relevant real-life problems: scheduling, route planning, solving optimal strategies for games, inventory control, investment problems, machine repair, text processing, DNA sequence matching, stopping problems, airplane/rocket flight path optimization, minimum time/effort problems, optimal fishery management, saving/consumption optimization, optimization of water allocation in hyrdrology, optimal feedback controllers for plants and regulator problems and so on.

    The models that are examined are:

    • Optimal control problem. The task is to find a control function that is admissible, i.e. satisfies some control constraints, that minimizes a given cost function, subject to (usually first-order) differential equation constraints.
    • Calculus of variations problem. Find twice continuously differentiable function with given boundary conditions that minimizes an integral functional over a fixed time horizon. The famous example is the Brachistochrone problem, where we seek shortest time curve, or curve of fastest descent, for a ball to slide down under uniform gravitational field. The problem was posed by Johann Bernoulli in 1696, and solved soon after that by Newton.
    • Dynamic Programming (DP) problem. Here the problem is formulated in a form that can be solved numerically using the recursive backward (or forward) dynamic programming algorithm. We will study dynamic, both deterministic and stochastic optimization problems in detail, and derive various DP algorithms for solving them. 

    Course details

    The course consists of 12 lectures, and 12 exercise classes. There are 10 rounds of homework assignment, as well as 10 rounds of student presentation assignments.

    Lecturer: Jonas Tölle

    Assistant: Otto Schultz

    Teaching: Online lectures (Zoom), online exercise classes (Zoom)

    Lectures:  Wednesday, 12:15 – 14:00; Thursday, 10:15 – 12:00

    https://aalto.zoom.us/j/69760170485

    Lecture videos: The online lectures are recorded and uploaded to Aalto Panopto and can be accessed here.

    Exercises: Wednesday, 14:15 – 16:00; Friday, 10:15 – 12:00

    https://aalto.zoom.us/j/67063812250?pwd=YTJhbE9ub21yYUNPZEg0VmZFQkNGQT09

    The online exercise sessions are not recorded.

    Assessment methods: Exam, homework assignments, presentation assignments

    Grading scale: 0–5

    Completing the course gives 5 ECTS credits. See Guide to the Course for more details on how to complete the course.

    Study material: Lecture slides and exercises are based on the material in the following books.

    • D. E. Kirk: Optimal Control Theory. Prentice Hall, 1970 (2004).
    • D. P. Bertsekas: Dynamic Programming and Optimal Control, vol 1 (and vol 2). Athena Scientific, 1995.
    • M. L. Kamien and N. L. Schwartz: Dynamic Optimization - The calculus of variations and optimal control in economics and management, 2nd edition. North Holland, 1991.

    The course books can be found as pdf-versions with a Google search. It is not necessary to read the books, but they can provide additional material for those who are interested. For example, they contain more problems and solutions.

    Language of instruction: English

    Prerequisites: 1st and 2nd years math, recommended MS-C2105 Introduction to Optimization (or equivalent)

    Weekly schedule (subject to change)

    The lectures and exercises follow the following schedule:
    1. Introduction, calculus of variations, Euler equation (Week 1).
    2. Calculus of variations, transversality conditions (Week 2).
    3. Optimal control (Week 2, Week 3).
    4. Pontryagin minimum principle, infinite time horizon in calculus of variations (Week 3).
    5. Minimum time and control-effort problems (Week 4).
    6. Dynamic Programming (DP), discrete time/state problems (Week 4).
    7. DP applications (Week 5).
    8. Continuous time problem, HJB equation (Week 6).
    9. Discounted problems, numerical methods (self-study).

    • forum icon
      Announcements Forum
    • forum icon
      Discussion forum

      Forum for discussing course content, lecture and homework material.

      We will monitor the forum and offer help to questions when possible.

      Posts violating Aalto University's code of conduct will be removed.

      Posting of solutions to homework exercises is prohibited!

  • Guide to the course

    Completing the course


    There are 2 problem sets with homework assignments per week, due on Sunday evening and Tuesday evening. These are based on the lecture material in question and the corresponding solved exercise problems. There are 0-6 presentation items in each problem set. The students are asked to prepare the problem sets before each exercise class and inform the assistant about their willingness to present their solutions. See Assignments for practical information about submitting, deadlines, and grading of homework assignments and the modalities for the students' presentation exercises.

    The course can be completed by participating in the exercise classes and taking the exam. The course can also be completed by just taking the exam.

    The students obtain 2 alternative grades on the scale 0–5:
    (1) just based on the result of their exam;
    (2) based on the result of their exam and their exercise submissions.
    From these two grades, the higher grade is automatically valid. The alternative grades are computed as follows:
    (1) The points in the exam (0–30) determine the final grade E;
    (2) The combined points are computed by scaling the points of the exam (0–30) linearly to 0–70, scaling the points of the homework exercises (0–h_max) linearly to 0–20, and by scaling the points for the presentation exercises 0–10 (0–p_max), and  summing these up. The combined points (0–100) determine the combined grade C. The homework problems make up 20 % of total points, and presentation problems make up 10 %, while the exam yields 70 %. However, for passing the course, at least 12 points in the exam are necessary.

    From the grades E and C, the higher grade is chosen. In both cases, 40% of the exam points (i.e., 12 points) are necessary for passing the course with grade 1.
    The precise thresholds for the particular grades are reported after the grading of the exams.

    Targeted learning outcomes


    After the course, the student has learned to perform the following tasks:
    • Solving continuous time problems using calculus of variations, including different end point conditions. Lectures 1-2 (Kirk Section 4).
    • Solving optimal control problems, including different end point conditions, and minimum principle. Lectures 3-5 (Kirk Section 5).
    • Solving infinite horizon problems. Lecture 4 (Bertsekas Vol 2, Section 1).
    • Solving discrete time/state problems using DP algorithm, including stochastic problems. Lectures 6-7 (Bertsekas Vol 1, Sections 1-5).
    • Solving continuous time problems using Hamilton-Jacobi-Bellman equation. Lecture 8 (Kirk Section 3.11).
    • Discounted problems and numerical methods, e.g. value- and policy iterations. Lecture 9 (Bertsekas Vol 1 Section 7).
    • Learning definitions for relevant concepts and terminology used in the course. These definitions are often asked in course exams.

    Exam

    Details TBA
    24.2.2022
    11.4.2022

  • Lectures

    Not available unless: You are a(n) Teacher

    The lecture slides may be updated as the course goes on.


  • Assignments

    Not available unless: You are a(n) Teacher

    The exercises may be updated as the course goes on.


    There are 10 rounds of exercises. Two types of problems yield points towards the course grade in grading scheme 2 (scheme 1 is 100 % exam). Homework problems make up 20 % of total points, and presentation problems make up 10 %, while the exam yields 70 %. In addition, there are problems solved in class, teacher demo problems, and self-study problems for which the solutions are handed for the student to familiarize with the topics.  

    For a few problems, including the large homework problem on round 6, Matlab is needed. Aalto students can get a free Matlib licence from https://download.aalto.fi/student/. For those unfamiliar with Matlab, the following link may be useful (although no guarantees are given): https://math.aalto.fi/opetus/MatOhjelmistot/2018syksy/Heikki/index.html

    Homework problems

    In each problem set, there is one homework problem worth 2 points. Round 6 is an exception; the problem is worth 4 points and is also anticipated to require twice the effort. Students submit their solutions in the Mycourses submission box 'Homework 1' etc. below, and the teaching staff grades them. Submission dedlines are at 22:00 o'clock on Sundays and Tuesdays. Deadlines are strict, late submissions are not accepted. The Mycourses system may have slight delays, so make sure to submit well ahead of the deadline!

    Grading: Full points = correct solution. Half of the points = partial solution on the right track. Zero points = otherwise.

    Format: Solutions are submitted individually. Include your name and student number. The solution must be submitted in a single PDF-file (except on round 6, when Matlab code is returned) in Mycourses. Files can be merged, for example, with, https://www.sodapdf.com/pdf-merge/. We recommend typing the solution on a computer, but clear hand-written solutions are also accepted. The solution must be presented in a clear and logical manner, with enough intermediate steps. If you're unable to solve some intermediate steps in your solution, consider using a placeholder notation, and continuing with the rest of the problem.

    Presentation problems

    There are 0-6 presentation items in each problem set. If a problem has subsections, they're presented by different students. Long problems are sometimes split into two. Every presentation item has the same weight. Correct solutions are posted on Mycourses, and students prepare to explain them before the exercise session.

    Grading: Points are awarded to those willing to present the solution in class, and able to do so if selected to present. A good effort with most parts successfully explained is enough. Students mark their willingness in Mycourses choice item 'Presentation 1' etc. below. Deadlines for marking are 15 minutes before the exercise session (14:00 on Wednesday, 10:00 on Friday). Deadlines are strict, late markings are not accepted. The Mycourses system may have slight delays, so make sure to mark your answer well ahead of the deadline!

    Format: The presenting students are selected 'randomly' after the deadline. The task is to explain the model solution. Therefore, students should use the model solution as the backbone of their presentation. If the model solution is very short, or important intermediate steps are missing, the student should fill in enough steps, so that there are no big leaps of faith. Whether the student uses the provided model solution document in the presentation is up to the student, as long as the presentation is easy to follow, follows the model solution, and the student's own contribution is clearly visible. This is important, so that other students can easily follow and compare their own work to the presented version. 

    When no or very few intermediate steps are needed, annotating the model solution document is probably the easiest option. There is an annotation example below (Round 1 problems with annotation examples), so you can get an idea about how annotations might look like. When adding annotations to the solution PDF, avoid using black colour, as it doesn't stand out. For typed text and equations, you can alternatively use a light background colour. If you have equations on an auxiliary document, it is advisable to 'letter' them in suitable groups (as in '(A)' for the first group), and add the letters to the relevant locations in the main document( as in 'equations (A)'), so that it's easy to follow the presentation.

    Starting from Round 3, a PowerPoint version of the model solution document is provided to make adding annotations, text, images and equations as easy as possible. You can type equations in PowerPoint either by inserting an equation, or using the MathType module. More detailed instructions are given in Typing equations for student presentations below, that follows and extends on the demonstration in the third exercise session.

    Examples of tools you can use for annotating PDFs:

    • Microsoft Edge browser (free)
    • Adobe Acrobat Reader (free)
    • PDF-XChange Editor (from download.aalto.fi/student/)
      You can use PDF-XChange to export to PowerPoint
    • MathType for typing equations in PowerPoint or Word (from download.aalto.fi/student/)
      Accepts TeX-input as well, in the format $equation$



    List of deadlines (subject to change)

    Presentation 1: Friday 14.1.2022 at 10:00
    Homework 1: Sunday 16.1.2022 at 22:00

    Presentation 2: Friday 21.1.2022 at 10:00
    Homework 2: Sunday 23.1.2022 at 22:00

    Homework 3: Tuesday 25.1.2022 at 22:00
    Presentation 3: Wednesday 26.1.2022 at 14:00
    Presentation 4: Friday 28.1.2022 at 10:00
    Homework 4: Sunday 30.1.2022 at 22:00

    Homework 5: Tuesday 1.2.2022 at 22:00
    Presentation 5: Wednesday 2.2.2022 at 14:00
    (No presentation: Friday 11.2.2022 at 10:00)
    (No deadline: Sunday 13.2.2022 at 22:00)

    (No deadline: Tuesday 8.2.2022 at 22:00)
    (No presentation: Wednesday 9.2.2022 at 14:00)
    Presentation 6: Friday 11.2.2022 at 10:00
    Homework 6: Sunday 13.2.2022 at 22:00

    Presentation 7: Wednesday 16.2.2022 at 14:00
    Homework 7: Thursday 17.2.2022 at 22:00 (updated)
    Presentation 8: Friday 18.2.2022 at 10:00
    Homework 8: Sunday 20.2.2022 at 22:00

    Homework 9: Tuesday 22.2.2022 at 22:00 (confirmed)
    Course exam: Thursday 24.2.2022 at 16:30 - 19:30

    Retake exam: Monday 11.4.2022 at 09:00 - 12:00

  • Lecture videos

    Not available unless: You are a(n) Teacher

    Lecture 1

    Lecture 4


    Lecture 5

  • Exam (11.4.2022, 9.00-13.00)

    Not available unless: You are a(n) Teacher

    Contact person: David Radnell <david.radnell@aalto.fi>, 0503038065


    Exam Paper and Return Box


    • At the start time of the exam, the exam papers (in pdf) will be available on this page.
    • The exam return box is below.
    • Please read the instructions below well ahead of the actual exam.
    • The exam will be available in: English.