Determining the optimal number of folds to use in a K-fold cross-validation: A neural network classification experiment

Figures & data

Figure 1. A schematic of a 5-fold cross-validation.

Figure 2. A schematic of a single neuron in neural network.

Figure 3. A schematic of neuron $\mathit{k}$ at the output layer of a multi-layer network.

Figure 4. A schematic of neuron $\mathit{j}$ at the hidden layer of a multi-layer network

Table 1. Parameters for the two-class normal mixture data

Class	Proportions	Mean Vectors	Covariance Matrix	Size
${\omega }_1$	${\pi }_1=0.4$ ${\pi }_2=0.3$ ${\pi }_3=0.3$	${\mu }_1=\left(\begin{array}{c}6\\ 7\end{array}\right)$ ${\mu }_2=\left(\begin{array}{c}7\\ 7\end{array}\right)$ ${\mu }_3=\left(\begin{array}{c}4\\ 9\end{array}\right)$	${\mathrm{\Sigma }}_1=\left(\begin{array}{cc}2.5& 0.9\\ 0.9& 1.4\end{array}\right)$ ${\mathrm{\Sigma }}_2=\left(\begin{array}{cc}0.3& 0\\ 0& 0.3\end{array}\right)$ ${\mathrm{\Sigma }}_3=\left(\begin{array}{cc}1.8& -0.2\\ -0.2& 0.4\end{array}\right)$	500
${\omega }_2$	${\pi }_1=0.4$ ${\pi }_2=0.6$	${\mu }_1=\left(\begin{array}{c}2\\ 2\end{array}\right)$ ${\mu }_2=\left(\begin{array}{c}2\\ 6\end{array}\right)$	${\mathrm{\Sigma }}_1=\left(\begin{array}{cc}1& -0.7\\ -0.7& 2.3\end{array}\right)$ ${\mathrm{\Sigma }}_2=\left(\begin{array}{cc}2& 0.8\\ 0.8& 1.7\end{array}\right)$	500

Figure 5. A plot of the number of hidden neurons and their respective error rate

Figure 6. A plot of the network training error rates without early stopping for 1000 folds.

Table 2. Closest error rates without early stopping

	Error rate	Standard error	90% confidence interval
Testing set	0.0503	0.00062	[0.04928, 0.05132]
K = 10 folds	0.0503	0.00014	[0.05007, 0.05053]
K = 15 folds	0.0503	0.00072	[0.04912, 0.05148]
K = 19 folds	0.0503	0.00078	[0.04902, 0.05158]
K = 16 folds	0.0502	0.00076	[0.04905, 0.05155]
K = 20 folds	0.0502	0.00082	[0.04885, 0.05155]

Figure 7. A plot of the network training error rates with early stopping for 1000 folds

Table 3. Closest error rates with early stopping

	Error rate	Standard error	90% confidence interval
Testing set	0.0807	0.00104	[0.07899, 0.08241]
K = 20 folds	0.0803	0.00093	[0.07877, 0.08183]
K = 13 folds	0.0802	0.00082	[0.07885, 0.08155]
K = 10 folds	0.0801	0.00088	[0.07865, 0.08155]
K = 15 folds	0.0801	0.00080	[0.07878, 0.08142]
K = 19 folds	0.0801	0.00084	[0.07872, 0.08148]

Data availability statement

The dataset used in this study was an artificial two-class normal mixture dataset generated using the MATLAB programming language version 9.2 (R2017a), with the MATLAB codes provided in the Appendix section of this article.