Uni-Logo

Deep Learning and Hyperparameter Optimization

The first Seminar will be held on 24th October!

The seminar language will be English (even if everyone is German-spoken, to practice presentation skills in English). The first meeting takes place on October 24th.

For projects, have a look at our open projects site.

Background

Many algorithms have free hyperparameters impacting the performance for a given application (e.g., numerical thresholds or discrete algorithm flags). As the setting of these parameters can have drastic effects, algorithm designers optimize them in practice, often manually which is a tedious, and time-consuming task. A promiment example of this are neural networks, which constitude state-of-the-art methods in various fields including computer vision, image recognition, and artificial intelligence, if properly tuned.

In this seminar, we will mainly study Bayesian optimization, a general framework often used for hyperparameter optimization, and its relationship to deep learning. This includes the application of Bayesian optimization to optimize the hyperparameter of neural networks, but also the use of recent advances in neural networks used to improve Bayesian optimization in general.

During the semester, you will learn about Bayesian optimization, and its application to hyperparameter optimization in general. At some point (e.g., during your MSc thesis), you will likely have to optimize the parameters of an algorithm you applied to a new domain, or developed yourself. The knowledge accquired during this seminar equip you with neccessary tools to successfully accomplish this.

In our group, we are actively working on both Deep Learning and Bayesian optimization. Both are very active and interesting fields with many open research questions. So if you are looking for a master project, and/or masters thesis, let us know.

Procedure

We will meet weekly (Monday 10:15-11:45, in building 051, room SR 00-031) to discuss research papers from the list of available papers below. Every week, one student presents a paper and leads the following discussion. All other participants read the paper and submit a summary plus 3 questions due the Friday before. The presenter will have access to the questions and is expected to take them into account during the presentation. All participants will discuss the paper, its merits, and limitations. This discussion will, in part, be guided by the questions submitted by the participants. At the end of the week the presenter hands in two page report on the paper.

The final grade takes the oral presentation and the written report into account.

Besides the seminar topic, you will learn several skills necessary not only in academia:

  • read and understand research papers
  • assesing the strengths and weaknesses
  • oral presentation in front of your peers
  • discussion with your peers
  • high level summary of research with which you are not intimately familiar.

What to put into the final report?

In a nutshell, we think of this report as a detailed summary of the paper you presented that also covers points that would come up in a research discussion about the paper. (We say "paper", here even though, you are not restricted to only write about the paper you present.)

E.g., some questions that should be discussed at some point in the report next to a detailed summary are the following:

  • What is the paper's main contribution and why is it important?
  • How does it relate to other techniques in the literature?
  • What are strong and what are weak points about the paper?
  • What would be interesting follow-up work? Any possible improvements in the methods? Any further interesting applications?
  • Is the code/data available online? Does it run off-the-shelf? If not, what problems are there with running it? (You should only put a limited amount of time into this; not longer than a full work day.)

Formatting and length of the final report

Final reports have to be typeset in LaTeX (sorry, but you were warned). We will use the formatting guidelines and electronic templates from the AI conference IJCAI. Reports don't have to be long (you already wrote all the paper summaries); 2 pages in IJCAI style are appropriate. Do not go beyond 4 pages - you might not be able to include everything you would like to include, but that is common in academic writing.

List of available papers

For questions, please email us: kleinaa@cs.uni-freiburg.de, sfalkner@cs.uni-freiburg.de fh@cs.uni-freiburg.de