Deep reinforcement learning (RL) algorithms are a powerful class of methods for optimizing sequential decision making and control problems, and are the driver behind many real-world applications. Yet, they are also very sensitive to their hyperparameters, such as the discount factor or the learning rate. This leads to considerable uncertainty with respect to optimal use. Setting these hyperparameters is a notoriously difficult task. At the same time, it is crucial for obtaining state-of-the-art performance. This translates into multiple costly training runs to find a well-performing agent. The problem is exacerbated in deep RL application scenarios that only allow for small amounts of data to be collected, such as in many biomedical applications. For example in the optimization of drug dosage schedules in cell cultures, the elapsed time between collected data points with meaningful variability is typically very large, such that only few data points can realistically be collected within a given time frame. Consequently, we aim to develop automated approaches for setting the hyperparameters of deep RL approaches in a sample-efficient manner for small datasets, thus reducing uncertainty.

Student: Asif Hasan

Deep reinforcement learning models tend to overfit to early experiences.There are many potential aspects that could be the root cause. In particular, deep learning tends to greedily follow the learning signal. However, in settings such as reinforcement or even continual learning, this signal changes over time, requiring more plasticity of the learning process than classical supervised settings. This project studies a novel approach to regularization to preserve plasticity in the model and explores the impact in various learning settings.

Student: Philipp Bordne

Traditional methods for Algorithm Configuration (AC) typically predict fixed parameter settings, yet research has shown that dynamically adapting parameters over time can significantly improve performance. This insight has led to the development of many handcrafted heuristics. To better respond to changing optimization dynamics, automatic Dynamic Algorithm Configuration (DAC) has been developed, with most current approaches relying on online Reinforcement Learning (RL). However, online RL is resource-intensive and, due to its need for extensive interactions with the environment, is not feasible for certain domains. Offline RL mitigates these challenges by training on pre-collected datasets, although it introduces new challenges, such as distributional divergence. This work explores the application of offline RL for DAC across various domains. We demonstrate that offline RL can effectively adapt the learning rate for Stochastic Gradient Descent (SGD).

Students: Leon Gieringer & Janis Fix

Model-based RL has shown tremendous sample efficiency and applicability in a broad variety of problem domains. However, typical benchmarks in which these models are being evaluated ignore a crucial aspect of real world scenarios -- partial observability. This work aims at understanding the impact of partial observability on model-based agents and tries to shed light on how world-models of partially observable worlds encode the underlying environment state.

Student: Sai Prasanna

Contextual reinforcement learning has shown to be a viable avenue to train generalizable agents by providing meta-information about the environments dynamics (such as the weight of a robot or the gravity of a planet). However, this setting has so far only been studied extensively in the online RL and model-based RL settings. Offline RL promises an alternative in which well performing policies are learned purely from existing datasets. In this work we set out to study the generalization capabilities of offline RL agents, when providing them with context information about the environment at hand.

Student: Rachana Tirumanyam