Simultaneous Ensemble Generation and Hyperparameter Optimization for Regression

David Roschewitz, Kurt Driessens, Pieter Collins

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

Abstract

The development of advanced hyperparameter optimization algorithms, using e.g. Bayesian optimization, has encouraged a departure from hand-tuning. Primarily, this trend is observed for classification tasks while regression has received less attention. In this paper, we devise a method for simultaneously tuning hyperparameters and generating an ensemble, by explicitly optimizing parameters in an ensemble context. Techniques traditionally used for classification are adapted to suit regression problems and we investigate the use of more robust loss functions. Furthermore, we propose methods for dynamically establishing the size of an ensemble and for weighting the individual models. The performance is evaluated using three base-learners and 16 datasets. We show that our algorithms consistently outperform single optimized models and can outperform or match the performance of state of the art ensemble generation techniques.
Original languageEnglish
Title of host publicationArtificial Intelligence
Subtitle of host publicationBNAIC 2017
EditorsB. Verheij, M. Wiering
PublisherSpringer
Pages116-130
Volume823
ISBN (Electronic)978-3-319-76892-2
ISBN (Print)978-3-319-76891-5
DOIs
Publication statusPublished - 2017

Publication series

SeriesCommunications in Computer and Information Science
Volume823

Keywords

  • Bayesian optimization
  • Hyperparameter optimization
  • Ensemble generation
  • Regression

Cite this

Roschewitz, D., Driessens, K., & Collins, P. (2017). Simultaneous Ensemble Generation and Hyperparameter Optimization for Regression. In B. Verheij, & M. Wiering (Eds.), Artificial Intelligence: BNAIC 2017 (Vol. 823, pp. 116-130). Springer. Communications in Computer and Information Science, Vol.. 823 https://doi.org/10.1007/978-3-319-76892-2_9