An Adaptive Neural Network Regression Method for Structure Identification

Abstract

This article reports a study on a flexible neural network regression method within the functional analysis of variance framework that aims to adapt to the underlying structure of the target function. We develop a novel penalization scheme where a concept of node impurity is introduced in the neural network framework. The node impurity in neural networks represents the homogeneity of the effects of the inputs on the node. We first define the effect of individual input on node and in turn, measure the node impurity based on the effects of inputs on node. We adopt the sum of node impurities as a penalty function whose usage makes the connections from inputs to nodes sparse, which improves estimation accuracy by reducing unnecessary complexity and enables data-adaptive structure identification. Our method takes into account of a large parameter space of the networks ranging from a fully-connected structure to sparsely connected structures. Among possible node connection structures, an optimal model is selected based purely on observed data. Numerical studies based on simulated and real datasets show that the proposed method performs well in identifying the inherent structure of the regression function and produces good estimation accuracy. Supplementary materials for this article are available online.

KEYWORDS:

• Functional analysis of variance
• Multivariate regression
• Neural network
• Node impurity
• Penalization

Supplemental Materials

Appendix: Definition of B-spline activation functions mentioned in Section 2.1 and numerical experiments to evaluate the coordinate descent algorithm and to investigate variable selection performance of the proposed method. (Appendix.pdf)

Source code: The R code used for the simulation examples presented in this article. (Source code.zip)

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

The research of Ja-Yong Koo was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (NRF2018R1D1A1B07049972, RS-2023-00219212 and RS-2023-00253020) and by a Korea University Grant (K2109351). The research of Kwan-Young Bak was supported by the Basic Science Research Program through NRF funded by the Ministry of Education, Science and Technology (RS-2022-00165581).