Model averages sharpened into Occam’s razors: Deep learning enhanced by Rényi entropy

Abstract

Ensemble methods of machine learning combine neural networks or other machine learning models in order to improve predictive performance. The proposed ensemble method is based on Occam’s razor idealized as adjusting hyperprior distributions over models according to a Rényi entropy of the data distribution that corresponds to each model. The entropy-based method is used to average a logistic regression model, a random forest, and a deep neural network. As expected, the deep leaning machine more accurately recognizes handwritten digits than the other two models. The combination of the three models performs even better than the neural network when they are combined according to the entropy-based method or according to methods that average the log odds of the classification probabilities reported by the models. Which of the best ensemble methods to choose for other applications may depend on the loss function that quantifies prediction performance and on a robustness consideration.

Keywords:

• Big data
• data science
• deep learning
• deep neural network
• model averaging
• model predictive distribution
• predictivism
• sharpened prior distribution

Acknowledgments

I thank the anonymous referee for guidance on how to improve the clarity of this paper.

Additional information

Funding

This research was partially supported by the Natural Sciences and Engineering Research Council of Canada (RGPIN/356018-2009).