The balance property in neural network modelling

Figures & data

Table 1. Comparison of the homogeneous model (null model), the logistic regression model, the early stopping FNN and the GLM improved/regularised FNN.

	In-sample	Out-of-sample	Balance
	learning loss on $\mathcal{D}$	test loss on $\mathcal{T}$	property (4(4) $\begin{array}{rl}\frac{1}{n}\sum _{i=1}^n{Y}_i& =\frac{1}{n}\sum _{i=1}^n{\hat{p}}^{\mathrm{M}\mathrm{L}\mathrm{E}}\left({\mathit{x}}_i\right)\\ & =\frac{1}{n}\sum _{i=1}^n\sigma \left({\hat{w}}_0^{\mathrm{M}\mathrm{L}\mathrm{E}}+\sum _{j=1}^q{\hat{w}}_j^{\mathrm{M}\mathrm{L}\mathrm{E}}{x}_{i,j}\right),\end{array}$(4) )
(1) Homogeneous model $\overline{p}$	0.3975	0.4070	5.007276%
(2) Logistic regression model ${\hat{p}}^{\mathrm{M}\mathrm{L}\mathrm{E}}$	0.3808	0.3901	5.007276%
(3a) Early stopping FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}}$, seed 1	0.3716	0.3813	5.144375%
(3b) Early stopping FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}}$, seed 2	0.3707	0.3810	4.803005%
(3c) Early stopping FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}}$, seed 3	0.3720	0.3824	4.915926%
(4a) Regularised FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}+}$, seed 1	0.3711	0.3810	5.007276%
(4b) Regularised FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}+}$, seed 2	0.3704	0.3806	5.007276%
(4c) Regularised FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}+}$, seed 3	0.3712	0.3815	5.007276%

Figure 1. Preliminary analysis exploring the early stopping rule: model fitting on the training data ${\mathcal{D}}_0$ (in upper graph) and tracking over-fitting on the validation data $\mathcal{V}$ (in lower graph); note that this is a standard output in Keras which (unfortunately) drops the factor 2 from the loss function (7(7) $(\mathit{w},\theta )\mapsto \mathcal{L}(\mathit{w},\theta ;\mathcal{D})=-\frac{2}{n}\ell (\mathit{w},\theta ;\mathcal{D}).$(7) ), thus, the y-axis needs to be scaled with 2.

Figure 2. (lhs) Balance property (4(4) $\begin{array}{rl}\frac{1}{n}\sum _{i=1}^n{Y}_i& =\frac{1}{n}\sum _{i=1}^n{\hat{p}}^{\mathrm{M}\mathrm{L}\mathrm{E}}\left({\mathit{x}}_i\right)\\ & =\frac{1}{n}\sum _{i=1}^n\sigma \left({\hat{w}}_0^{\mathrm{M}\mathrm{L}\mathrm{E}}+\sum _{j=1}^q{\hat{w}}_j^{\mathrm{M}\mathrm{L}\mathrm{E}}{x}_{i,j}\right),\end{array}$(4) ) of the early stopping FNN over 50 different seeds (starting points), the orange horizontal line shows the balance property of 5.007276%; (middle) in-sample learning losses on $\mathcal{D}$ and (rhs) out-of-sample test losses on $\mathcal{T}$ of the early stopping FNN (left box plots in graphs) and the GLM improved/regularised FNN (right box plots in graphs) over the 50 different seeds (starting values of the SGD algorithm).

Table 2. Comparison of the homogeneous model (null model), the logistic regression model, the early stopping FNN and the GLM improved/regularised FNN, shrinkage regularised versions for different tuning parameters $\eta \in \{10,100,250,1000\}$.

	In-sample	Out-of-sample	Balance
	learning loss on $\mathcal{D}$	test loss on $\mathcal{T}$	property (4(4) $\begin{array}{rl}\frac{1}{n}\sum _{i=1}^n{Y}_i& =\frac{1}{n}\sum _{i=1}^n{\hat{p}}^{\mathrm{M}\mathrm{L}\mathrm{E}}\left({\mathit{x}}_i\right)\\ & =\frac{1}{n}\sum _{i=1}^n\sigma \left({\hat{w}}_0^{\mathrm{M}\mathrm{L}\mathrm{E}}+\sum _{j=1}^q{\hat{w}}_j^{\mathrm{M}\mathrm{L}\mathrm{E}}{x}_{i,j}\right),\end{array}$(4) )
(1) Homogeneous model $\overline{p}$	0.3975	0.4070	5.007276%
(2) Logistic regression model ${\hat{p}}^{\mathrm{M}\mathrm{L}\mathrm{E}}$	0.3808	0.3901	5.007276%
(3a) Early stopping FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}}$, seed 1	0.3716	0.3813	5.144375%
(4a) Regularised FNN ${\hat{p}}^{\mathrm{S}\mathrm{G}\mathrm{D}+}$, seed 1	0.3711	0.3810	5.007276%
(5a) Shrinkage regularised with $\eta =10$	0.3718	0.3820	5.207727%
(5b) Shrinkage regularised with $\eta =100$	0.3740	0.3833	5.099347%
(5c) Shrinkage regularised with $\eta =250$	0.3759	0.3849	5.049608%
(5d) Shrinkage regularised with $\eta =1000$	0.3820	0.3911	5.012383%