Performance analysis of hyperparameter optimization methods for ensemble learning with small and medium sized medical datasets

Abstract

The accuracy of the various classifiers depends mainly on good hyper-parameter and consequently on the scheme (hyper-parameter tuning algorithm) adopted to estimate these values. Currently, the hyper-parameter tuning for Ensemble classifiers (which involves a number of hyper-parameters) is receiving a lot of attention. This paper aims to analyze the effectiveness of three optimization methods: Grid Search (GS), Random Search (RS) and Bayesian Optimization (BO) on small and medium sized medical datasets to select a set of optimal hyper parameters for an ensemble classifier (here we used decision tree ensemble using AdaBoost). 5 fold CV wasemployed to evaluate the generalization performance.

Subject Classification:

• Primary 62H30

Keywords:

• Bayesian Optimization
• Grid Search
• Random Search
• Ensemble classifier
• Hyper-parameter Optimization