Uni-Logo

Automated Parameter Tuning and Algorithm Configuration

Seminar at the 31st July will start at 14:45.

The seminar language will be English (even if everyone is German-spoken, to practice presentation skills in English). First meeting: May 8th.

Background

Many algorithms have free parameters that can be modified to improve performance in practice (e.g., numerical thresholds or discrete algorithm flags). The setting of these parameters often has a drastic impact, and thus algorithm designers optimize their parameters in practice, often manually (a tedious and time-consuming task). In this seminar, we will study automated approaches for this algorithm configuration problem. We will discuss both foundations of algorithmic procedures for solving the general algorithm configuration problem and their applications to optimize algorithm parameters in practice.

Foundations include concepts from statistics (experimental design, statistical tests), machine learning (regression, stochastic processes), optimization (mixed continuous/discrete optimization, stochastic optimization, global optimization), and artificial intelligence (local search, population-based methods).

Possible applications span algorithms in most areas of computer science, including, e.g.: machine learning, formal verification of hardware and software, AI planning, computer graphics, compiler optimization, database query optimization, numerical optimization, and algorithm engineering. They also include algorithms in various related areas, ranging from engineering to operations research.

Likely, at some point (e.g., during your MSc thesis) you will need to optimize the parameters of an algorithm you developed. In this seminar, you will learn how to do so in a scientific, convenient, and effective manner.

Algorithm configuration is also a young and dynamic field of research, full of interesting questions for practica, semester projects, MSc theses, and so on. (If you're interested in any of these during/after the course, let me know.)

Dynamics

We will meet weekly (Thursday 14:00-16:00 c.t., in building 052, room HS 02-017) to discuss research papers from the following list of available papers. For each paper, one participant will have the lead and give a presentation. All participants will read each paper and submit a summary and 3 questions about it; these will be due 2 days before the presentation (to give the presenter time to take them into account). After the presentation, all participants will discuss the paper, and its merits and limitations. This discussion will in part be guided by the questions submitted by the participants.

Towards the end of term, each participant will hand in a (relatively brief) written report, typically on the topic of the paper they presented (although a related subject is a possibility). Final marks will be formed using a combination of marks for the oral presentation and written report quality of summaries and questions submitted, and class participation.

Overall, next to the seminar topic, you will learn several soft-skills necessary in academia: to read and understand research papers and assess their strengths and weaknesses; to orally present in front of your peers; to discuss with your peers; and to summarize on a high level a piece of research you are not intimately familiar with.

Resources

Slides from kickoff meeting
List of available papers

What to put into the final report?

In a nutshell, we think of this report as a detailed summary of the paper you presented that also covers points that would come up in a research discussion about the paper. (We say "paper" here even though technically, you are not restricted to write about the paper you present.)

E.g., some questions that should be discussed at some point in the report next to a detailed summary are the following:

  • What is the paper's main contribution and why is it important?
  • How does it relate to other techniques in the literature?
  • What are strong and what are weak points about the paper?
  • What would be interesting follow-up work? Any possible improvements in the methods? Any further interesting applications?
  • Is the code/data available online? Does it run off-the-shelf? If not, what problems are there with running it? (You should only put a limited amount of time into this; not longer than a full work day.)

Formatting and length of the final report

Final reports have to be typeset in LaTeX (sorry, but you were warned). We will use the formatting guidelines and electronic templates from the AI conference IJCAI. Reports don't have to be long (you already wrote all the paper summaries); 2 pages in IJCAI style are appropriate. (It's OK if it's a bit longer, but please don't go beyond 4 pages - you might not be able to include everything you would like to include, but such is life in academic writing.)

For questions, please send us email: lindauer@cs.uni-freiburg.de, fh@cs.uni-freiburg.de