Profile – Machine Learning Lab

Professor

Frank Hutter

Postal address

Institut für Informatik
Albert-Ludwigs-Universität Freiburg
Sekretariat Hutter/Maschinelles Lernen
Georges-Köhler-Allee 074
79110 Freiburg, Germany

Fax

+49 761 203-74217

Office

Building 74, Room 00-017

+49 761 203-67741 (secretary)

Publications

I'm the Head of the Machine Learning Lab. I'm lucky enough to have an amazing team; all my Freiburg team members are linked in the menu on the top; Tübingen team members to come.

I'm currently holding an ERC Consolidator Grant on Deep Learning 2.0 and previously held ERC Starting and ERC PoC grants, as well as an Emmy Noether fellowship.

Information for students interested in projects, theses, or Hiwi positions

With machine learning being one of the hottest topic around, our small group is flooded with requests.
To make the process efficient, please do not email me directly, but follow the instructions posted under “working with us”.

My position as Hector Endowed Fellow at the ELLIS Institute Tübingen

I'm on temporary leave at the ELLIS Institute Tübingen. The ELLIS Institute is set to become a European Lighthouse of Machine Learning Excellence, and I’m thrilled to be part of it. Together with my new colleagues, and in close collaboration with my Freiburg team, I look forward to expanding my research on AutoML in the most impactful way possible. I will continue to hire in Freiburg, but now I also have open positions in Tübingen. Next to the obvious positions in AutoML, I’m also looking for research engineers, postdocs and PhD students who are excited to build and would like to dive deep into large language models (pretraining, fine-tuning, and the connection of AutoML to both of these, as well as hosting open LLMs for Europe).

Commentary on AI risks

Our commentary on AI risks (with Yoshua Bengio, Geoff Hinton and ~20 other top AI researchers) just got published in Science Magazine. As the only European coauthor, I'd like to stress that I'm against more regulation in Europe; we already have enough of this. Instead, Europe needs to catch up on AI, by making dramatically higher investments (into basic research, startups and much more compute resources). We have an amazing open source community in Europe and we need to embrace it, rather than regulate it away. Open source is Europe‘s only hope to get out of the current deep technological dependence on the US. AI will shape the future of humanity, one way or the other, and we need to get this right. With so much at stake, AI needs to finally become a top strategic priority for Europe.

For more on this commentary, please see my LinkedIn post.

Research focus

I focus on Automated Machine Learning (AutoML), an area in which I am proud to have a world leading group and to be amongst the most cited researchers worldwide. This includes research in the following areas, with some examples:

Efficient hyperparameter optimization (HPO)
- Bayesian optimization, multi-fidelity optimization, HPO benchmarks, Transfer HPO. E.g., we won the 2021 NeurIPS Blackbox optimization competition (warmstarting-friendly track).
Neural architecture search (NAS)
- Speedup techniques for NAS by one-shot models, network morphisms, multi-fidelity optimization, and meta-learning; tabular & surrogate NAS benchmarks
Meta-learning
- Learning to perform Bayesian inference, meta-learning for NAS + HPO, learning algorithm components, learning hyperparameter adaptation strategies
AutoML systems
- Auto-WEKA, Auto-sklearn, Auto-PyTorch. E.g., we won the first two international AutoML challenges by ChaLearn with Auto-sklearn.
Scientific experimentation
- Making empirical research more reproducible and codifying human experts' strategies to the point where an autonomous system can execute them
Auto-AI
- Algorithm configuration, algorithm selection and dynamic algorithm configuration; we've won many SAT competitions and AI planning competitions with these Auto-AI techniques
Explainable AutoML (xAutoML)
- Performance prediction and automated algorithm analyses. Here’s a (dated) Google tech talk I gave on Auto-AI + xAutoML

For more information on AutoML, please see our AutoML book (the first book on AutoML) or our NeurIPS 2018 tutorial (which had over 3000 attendees).

Along the way, I've become a deep learner, with additional interests in

Architectures, optimizers and regularization methods
Uncertainty
- Bayesian optimization with Bayesian neural networks (SG-MCMC)
- Prior-data fitted networks (PFNs), an entirely new way to perform Bayesian inference via meta-learning with Transformers
Applications of deep learning and AutoML
- We were the first to obtain competitive performance on EEG decoding with deep learning
- We dramatically improved RNA design using deep RL, meta-learning and joint NAS+HPO
- We were the first to achieve state-of-the-art performance on tabular data with deep learning, outperforming even properly-tuned gradient boosting. This opens a huge part of data science to the benefits of modern deep learning.

Affiliations

I'm a Hector-Endowed Fellow and PI at the ELLIS Institute Tübingen, a member of the Department of Computer Science at the Faculty of Engineering of the University of Freiburg, and head of the ELLIS unit Freiburg. I hold an ERC Consolidator Grant and previously held an ERC Starting Grant and an ERC PoC Grant from the European Research Council, as well as an Emmy Noether Grant from the German Research Foundation (DFG). I'm a former member of the Computer Science Department of the University of British Columbia (UBC), specifically of the Laboratory for Computational Intelligence (LCI) and the Bioinformatics and Empirical & Theoretical Algorithmics Laboratory (BETA). In addition to my full-time role at the University of Freiburg, I also consulted for the Bosch Center for AI (BCAI) as their Chief Expert for AutoML (2019-2023). I also was a machine learning consultant for Zynga Inc and am a co-founder of Meta-Algorithmic Technologies. I earned my PhD at UBC in 2009 and my Diplom (eq. MSc) at Darmstadt University in 2004.

I was the inaugural general chair of the AutoML conference in 2022 and 2023. I was program co-chair of ECML-PKDD 2020. I co-founded and co-organized the ICML workshop series on AutoML every year 2014 – 2021 (after which we turned it into the AutoML conference), the NeurIPS workshop series on meta-learning, and the Neural Architecture Search workshop at ICLR. I also co-founded and regularly co-organized the NeurIPS workshop series on Bayesian optimization. Please see http://automl.org/workshops for an up-to-date list of workshops I'm co-organizing.

Short bio

Frank Hutter is a Hector-Endowed Fellow and PI at the ELLIS Institute Tübingen, as well as Full Professor for Machine Learning at the University of Freiburg (Germany). Frank holds a PhD from the University of British Columbia (UBC, 2009) and a Diplom (eq. MSc) from TU Darmstadt (2004). He received the 2010 CAIAC doctoral dissertation award for the best thesis in AI in Canada, and with his coauthors, several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. He is a Fellow of EurAI and ELLIS, the director of the ELLIS unit Freiburg and the recipient of 3 ERC grants. Frank is best known for his research on automated machine learning (AutoML), including neural architecture search, efficient hyperparameter optimization, and meta-learning. He co-authored the first book on AutoML and the prominent AutoML tools Auto-WEKA, Auto-sklearn and Auto-PyTorch, won the first two AutoML challenges with his team, is co-teaching the first MOOC on AutoML, co-organized 15 AutoML-related workshops at ICML, NeurIPS and ICLR, and founded the AutoML conference as general chair in 2022 and 2023. In recent years, his focus has been on the intersection of foundation models and AutoML, including the first foundation model for tabular data, TabPFN, and improving pretraining and fine-tuning with AutoML.

Publications

2024

Ferreira, Fabio; Schlageter, Moreno; Rajan, Raghu; Biedenkapp, André; Hutter, Frank

One-shot World Models Using a Transformer Trained on a Synthetic Prior Inproceedings

In: NeurIPS 2024 Workshop on Open-World Agents, 2024.

Arango, Sebastian Pineda; Janowski, Maciej; Purucker, Lennart; Zela, Arber; Hutter, Frank; Grabocka, Josif

Ensembling Finetuned Language Models for Text Classification Inproceedings

In: NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024.

Hoo, Shi Bin; Müller, Samuel; Salinas, David; Hutter, Frank

The Tabular Foundation Model TabPFN Outperforms Specialized Time Series Forecasting Models Based on Simple Features Inproceedings

In: NeurIPS 2024 TRL Workshop, 2024.

Feuer, Benjamin; Schirrmeister, Robin Tibor; Cherepanova, Valeriia; Hegde, Chinmay; Hutter, Frank; Goldblum, Micah; Cohen, Niv; White, Colin

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks Inproceedings

In: 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.

Küken, Jaris; Purucker, Lennart; Hutter, Frank

Large Language Models Engineer Too Many Simple Features for Tabular Data Inproceedings

In: NeurIPS 2024 Third Table Representation Learning Workshop, 2024, (Oral Presentation).

Grazzi, Riccardo; Siems, Julien; Franke, Jörg K. H.; Zela, Arber; Hutter, Frank; Pontil, Massimiliano

Unlocking State-Tracking in linear RNNs through Negative Eigenvalues Inproceedings

In: NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning Workshop (M3L), 2024, (Oral Presentation).

Bhethanabhotla, Sathya Kamesh; Swelam, Omar; Siems, Julien; Salinas, David; Hutter, Frank

Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Models Inproceedings

In: NeurIPS 2024 TSALM Workshop, 2024, (Spotlight Presentation).

Müller, Andreas; Siems, Julien; Nori, Harsha; Salinas, David; Zela, Arber; Caruana, Rich; Hutter, Frank

GAMformer: Exploring In-Context Learning for Generalized Additive Models Inproceedings

In: NeurIPS 2024 TRL Workshop, 2024.

Sukthanker, Rhea Sanjay; Staffler, Benedikt; Hutter, Frank; Klein, Aaron

Large Language Model Compression with Neural Architecture Search Inproceedings

In: NeurIPS 2024 Workshop on Machine Learning and Compression, 2024.

Mallik, Neeratyoy; Janowski, Maciej; Hog, Johannes; Rakotoarison, Herilalaina; Klein, Aaron; Grabocka, Josif; Hutter, Frank

Warmstarting for Scaling Language Models Inproceedings

In: NeurIPS 2024 Workshop Adaptive Foundation Models, 2024.

Franke, Jörg K. H.; Hefenbrock, Michael; Koehler, Gregor; Hutter, Frank

Improving Deep Learning Optimization through Constrained Parameter Regularization Inproceedings

In: 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.

Sukthanker, Rhea Sanjay; Zela, Arber; Staffler, Benedikt; Klein, Aaron; Purucker, Lennart; Franke, Joerg K. H.; Hutter, Frank

HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models Inproceedings

In: 38th Conference on Neural Information Processing Systems (NeurIPS), DBT Track, 2024.

Helli, Kai; Schnurr, David; Hollmann, Noah; Müller, Samuel; Hutter, Frank

Drift-Resilient TabPFN: In-Context Learning Distribution Shifts on Tabular Data Inproceedings

In: 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.

Strangmann, Tobias; Purucker, Lennart; Franke, Jörg K. H.; Rapant, Ivo; Ferreira, Fabio; Hutter, Frank

Transfer Learning for Finetuning Large Language Models Inproceedings

In: NeurIPS 2024 Workshop on Adaptive Foundation Models, 2024.

Robertson, Jake; Schmidt, Thorsten; Hutter, Frank; Awad, Noor

A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes Inproceedings

In: Proceedings of the Seventh AAAI/ACM Conference on AI, Ethics, and Society (AIES-24), 2024.

Scheuer*, Dominik; Runge, Frederic*; Franke, Jörg K. H.; Wolfinger, Michael T.; Flamm, Christoph; Hutter, Frank

KinPFN: Bayesian Approximation of RNA Folding Kinetics using Prior-Data Fitted Networks Working paper

2024.

Arango, Sebastian Pineda; Janowski, Maciej; Purucker, Lennart; Zela, Arber; Hutter, Frank; Grabocka, Josif

Dynamic Post-Hoc Neural Ensemblers Inproceedings

In: Preprint, 2024.

Runge, Frederic; Hutter, Frank

Machine Learning for RNA Design: LEARNA Book Chapter

In: Churkin, Alexander; Barash, Danny (Ed.): RNA Design: Methods and Protocols, Chapter 5, pp. 63–93, Springer US, New York, NY, 1, 2024, ISBN: 978-1-0716-4079-1.

Shala, Gresa; Arango, Sebastian Pineda; Biedenkapp, André; Hutter, Frank; Grabocka, Josif

HPO-RL-Bench: A Zero-Cost Benchmark for HPO in Reinforcement Learning Inproceedings

In: Proceedings of the Third International Conference on Automated Machine Learning (AutoML 2024), ABCD Track, 2024, (Runner up for the best paper award).

Sukthanker, Rhea Sanjay; Krishnakumar, Arjun; Safari, Mahmoud; Hutter, Frank

Weight-Entanglement Meets Gradient-Based Neural Architecture Search Inproceedings

In: Proceedings of the Third International Conference on Automated Machine Learning (AutoML 2024), Methods Track, 2024.

Grazzi, Riccardo; Siems, Julien; Schrodi, Simon; Brox, Thomas; Hutter, Frank

Is Mamba Capable of In-Context Learning? Inproceedings

In: Proceedings of the Third International Conference on Automated Machine Learning (AutoML 2024), Methods Track, 2024.

Strack, Lukas; Safari, Mahmoud; Hutter, Frank

Towards Efficient Search for Customized Activation Functions With Gradient Descent Inproceedings