Bayesian regularized neural network decision tree ensemble model for genomic data classification

ABSTRACT

Machine learning techniques have been widely applied to solve the classification problem of highly dimensional and complex data in the field of bioinformatics. Among them, Bayesian regularized neural network (BRNN) became one of the popular choices due to its robustness and ability to avoid over fitting. On the other hand, Bayesian approach applied to neural network training offers computational burden and increases its time complexity. This restricts the use of BRNN in an on-line machine learning system. In this article, a Bayesian regularized neural network decision Tree (BrNdT) ensemble model, is proposed to combat high computational time complexity of a classifier model. The key idea behind the proposed ensemble methodology is to weigh and combine several individual classifiers and apply majority voting decision scheme to obtain an efficient classifier which outperforms each one of them. The simulation results show that the proposed method achieves a significant reduction in time complexity and maintains high accuracy over other conventional techniques.

Introduction

Bioinformatics is one of the interdisciplinary fields which develop algorithms and methods for analyzing biological data to produce meaningful information (Christopher et al. 2009). During past few decades, in the area of feature-based classification, binary classification problems have been studied extensively but multi category classification has its own importance. Research also shows that multi-class classification is much complex than binary classification and accuracy of classifier drops significantly as the number of classes increases. In literature, various multi-class classifier algorithms have been devised by researchers such as decision tree (DT) (Asria et al. 2016; Patil, Joshi, and Toshniwal 2010), artificial neural network (ANN) (Thein and Tun 2015), support vector machine (SVM) (Bazazeh and Shubair 2017; Barale and Shirke 2016; Megha. and Pareek 2016; Salama, Abdelhalim, and Zeid 2012), and Bayesian regularized artificial network (BRNN) (Burden and Winkler 2008).

Figure 3. Structure of majority voting ensemble.

Figure 4. Proposed ensemble framework for classifier.

Figure 5. Performance comparison of classification accuracy in various schemes.

Figure 6. Performance comparison of time complexity involved in various classifiers.

DT algorithm (Patil, Joshi, and Toshniwal 2010) is a simple and widely used classification scheme. The principle of DT classifiers is based on a series of test questions and conditions in a tree structure. In general, DTs are robust to outliers but due to their tendency to over fit, they are much prone to sampling errors. DTs do not offer good results if sampled training data is somewhat different than scoring or evaluation data. ANN model offers an attractive solution for a direct multi-category classification problem (Thein and Tun 2015; Mishra and Zaheeruddin 2010a, 2010b). Neural networks can also be adapted easily to produce continuous variables instead of discrete class labels. This helps to predict the level of the variables used rather than to classify the samples into two or more categories. However, conventional neural networks usually provide lower classification accuracy due to its tendency to get stuck in local minima.

Contrary to ANN, SVMs deliver a unique solution and is also known as a large-margin classifier. The major objective of SVM is to find a decision boundary between two classes that remains maximally far from any point in the training data set. Though, SVMs are inherently two-class classifiers but can also be extended to a task of multiclass classification. One of the major limitations of support vector approach lies in the choice of optimum kernel. Size of SVMs network also restricts its performance, both in training and testing phase. SVMs also offer high algorithmic complexity and extensive memory requirements due to its quadratic programming in large-scale tasks.

Bayesian regularized artificial neural networks (BRANNs) (Burden and Winkler 2008) are more robust than standard Neural Network models based on back-propagation algorithm. Bayesian regularization converts a nonlinear regression into a “well-posed” statistical problem mathematically through a process of a ridge regression. These models also provide a less complex validation process as compared to normal regression methods such as back propagation algorithm. Moreover, BRANN uses objective Bayesian criterions to stop training which eliminates the possibility of over train and overfit of the network. The major limitation of these networks is the calculation, storage and inversion of Hessian matrices during training phase.

Therefore, it is the need of research to design a time efficient algorithm, which not only offers high accuracy but also outperform other equivalent classifiers based on its training speed and computational complexity. Motivated by this research gap, in this paper, an effort has been initiated to design a Bayesian regularized neural network decision Tree (BrNdT) ensemble mode classifier.

The rest of the paper is organized as follows: Section 2 presents popular existing models with their basic principles. Section 3 describes proposed methodology used in paper. The results and evaluation are discussed in Section 4. Finally, conclusion and future work are given in Section 5.

Existing models

Bayesian regularized neural networks (BRNN)

Bayesian regularization is a mathematical technique which converts a non-linear regression into a statistical well posed problem. The Bayesian regularized neural network is robust in nature (Burden and Winkler 2008). The Bayesian scheme for neural networks is developed on the probabilistic interpretation of network parameters. Bayesian approach includes a probability distribution of network weights. Bayesian approach resolves the over fitting problem. The complex models are also penalized in Bayesian approach (Kayri 2016). BRNN is the linear combination of ANN and Bayesian methods to determine the optimal regularization parameters. It involves imposing specified prior distributions on the parameters of model. Figure 1 shows the dataflow diagram of BRNN algorithm. In Eq. (1), term is used to anticipate a better generalization and smooth mapping (Okut 2016).

(1)

Figure 1. Data flow diagram of BRNN algorithm.

whereis sum of squared estimation errors, M is ANN Architecture and denotes sum of squares of architecture weights.and are regularization parameters also called objective function parameters. The term represents the weight decay, where is the weight decay coefficient. To decrease the tendency of overfitting,should have smaller values.

Eq. (1) involves tradeoff between goodness of fit and model complexity. When then it produces a smoother network response at the expense of goodness of fit.

Decision tree (DT)

DT is a well-known classification method also known as Classification Tree. A DT comprises of a set of internal and leaf nodes. The internal nodes are linked with a splitting criterion, consisting of a splitting attribute and splitting predicates defined on that splitting attribute. The leaf nodes are labeled using a single class label.

DT has various advantages which makes its performance outstanding from several other algorithms. DT don’t require any input parameter, and its construction technique is relatively fast. Moreover the results are also interpretable (Asria et al. 2016). DTs also provide the scalability features for large datasets. DT can classify both numerical and categorical data.

Linear regression (LR)

Regression is one among the common techniques used for prediction. Regression is one of the popular techniques used for prediction. Linear regression performs prediction for dependent variable by determining relationship between independent and dependent variables (Mehra 2003). The graphical relationship between variables can be easily represented by regression model. Mathematical representation of Regression model can be described as follows (Weiss and Kulikowski 1991):

(2)

where i ranges from . T represents transpose of which is usedto determine the inner product between and.

By the combination of n in above equations, it can be written in the vector form as:

(3)

where is a dependent variable, and is an independent variable, represents regression coefficients, that are p-dimensional parameter vectors and represents errors term. Numerous attributes provided in the datasets are taken as independent variables and output (sick/healthy) is considered to be dependent variable.

Support vector machine (SVM)

Support Vector Machine is supervised learning algorithm suggested by Vapnik. SVM is used for classification and regression tasks (Salama, Abdelhalim, and Zeid 2012). It is a binary classifier which separates the data by creating a margin hyper-plane. The kernel function is used for separation of dataset which is non-linear in nature. The SVM is represented by the following:

(4)

where depicts the problem to be classified and represents maximum margin hyper plane. The data points are called support vectors.

In Table 1, d is a positive integer value and depicts the degree of kernel. Here σ is a real positive value.

Table 1. Types of kernel with its equations.

Kernel Type	Representation
Dot Product Kernel Polynomial kernel RBF kernel

Proposed methodology

Ensemble classifier

In data mining, each classifier performs prediction based on its supervised learning. In the field of machine learning, ensemble methods were first invented a decade back. The ensemble classifiers weigh several individual classifiers and then fuse them together in order to achieve obtain the results which transcend every individual classifier, and as shown in Figure 2. The ensemble classifier has better prediction accuracy and classification performance than individual classifiers (Polikar 2006).

The proposed research converges on the classifiers such as for instance DT, Bayesian regulatory neural network, and linear regression to form ensemble classifier.

Majority voting

The technique of classifying the unlabeled instances, based on the high frequency vote (or the highest number of votes) is coined as majority voting or plurality voting (PV). Three classifiers namely DT, Bayesian regularized neural network and linear regression combine to form majority voting scheme which is used to predict the class of unlabeled diabetes instances. Majority voting ensemble structure is depicted in Figure 3. Mathematically, it can be represented as (Rokar 2010).

(5)

Figure 2. Ensemble framework.

where the classification of the classifier is denoted by and is an indication function defined as

(6)

A crisp classification can be obtained in the case of probabilistic classifier by following formula

(7)

in which classifier is represented by and the probability of class c is represented by for an instance .

Algorithm for majority voting ensemble and probability of correct labeling

Require:

D: Training dataset

L: Learning algorithm

W: Labels of training dataset

N: Number of Learning algorithm used

d: Output of Classifier

Do n = 1:N

1. Initialize L_n for every D_n .

2. Compare W_n with d_n generated from L_n, and update vote.

3. Plurality results in a decision as a output from ensemble.

End

Selection of data base

In this paper three standard datasets are selected to verify robustness of the proposed scheme. The three datasets namely a) Pima Indian Diabetes, b) CpG island, c) Wisconsin Diagnostic Breast Cancer, represent three different real world classification problems available at UCI repository. The detail of datasets used is shown in Table 2.The datasets contain the feature space that distinguishes healthy and sick individuals into two classes. Each dataset has different various sets of attributes and data types. For every evaluation, constant proportion of training examples is randomly selected and verified on the testing set.

• (a) Pima Indian Diabetes data can be obtained from UCI repository of machine learning:(https://archive.ics.uci.edu/ml/datasets/pima+indians+diabetes)

Table 2. Dataset employed in the experiment.

Dataset	Train	Test	Attributes	Instances	Positive Instances	Negative Instances	Classes
Diabetes	460	308	9	768	268	500	2
CpG island	297	199	38	495	84	411	2
Breast Cancer	341	228	32	569	35	212	2

The database is created in National Institute of Diabetes and Digestive and Kidney Diseases. Several constraints were adopted on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage. The various attributes of Pima Indian Diabetes data set is listed in Table 3.

• (b) Cpg Island dataset methylation defines the status of human chromosomes. The dataset is referred from BioMed Central Bioinformatics (Christopher et al. 2009). The dataset has 38 attributes which belong to four distinct categories namely i) CGI specific attributes (including G + C content, CpG cluster p-value, Observed/Expected ratio), ii) Repetitive sequences (eg. Number and type of repetitive elements, iii) Evolutionary Conservation (e.g. PhaseCon content), iv) Physiochemical and Structural Properties of DNA (includes twist, tilt, roll, rise, shift, slide).

Table 3. Description of attributes of pima Indian diabetes dataset.

Attributes	Min/Max
Number of times pregnant	0/17
Concentration of plasma glucose in an oral glucose tolerance test in 2hours	0/199
Diastolic blood Pressure (mm Hg)	0/122
Thickness of triceps skin folds (mm	0/99
Serum insulin in 2hour (mu U/ml)	0/846
BMI (Body Mass Index) (weight in kg/(height in m^2)	0/67.1
Diabetes Pedigree Function	0.078/2.42
Age (years)	21/81

The attributes of second dataset are shown in Table 4.

• (c) Wisconsin Diagnostic Breast Cancer Data is made available at following link by William H. Wolberg University of Wisconsin Hospital.

  https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)

Table 4. Description of attributes for CpG Island dataset.

Attributes	Min/Max	Attributes	Min/Max	Attributes	Min/Max
CpG cluster value	0/1	TA content	0/1	Curvature	0/1
Observed/Expected value	0/1	AT content	0/1	Stacking energy	0/1
Average distance between CpGs	0/1	CA content	0/1	Turn	0/1
Standard Deviation	0/1	TG content	0/1	Twist	0/1
G + C content	0/1	AC content	0/1	Cleavage	0/1
Repetitive Element content	0/1	GT content	0/1	Bases per turn	0/1
Repetitive elements	0/1	AG content	0/1	Twist constraint	0/1
PhastCon content	0/1	CT content	0/1	Tilt constraint	0/1
PhastCon elements	0/1	CC content	0/1	Roll constraint	0/1
CG content	0/1	GG content	0/1	Shift constraint	0/1
GC content	0/1	GA content	0/1	Slide constraint	0/1
AA content	0/1	TC content	0/1	Rise constraint	0/1
TT content	0/1	Bending	0/1

It has two classes with 569 instances i.e. 357 benign and 212 malignant. It has 10 variables consisting of cellular characteristics computed from digitized image of breast fine needle aspirate. The standard error, mean and worst of these attributes were computed for every image, which results in 30 features. The real valued attributes collected from the following link and listed in Table 5. https://bigml.com/user/czuriaga/gallery/dataset/51794b29ce5680176800031e

Table 5. Description of attributes for wisconsin diagnostic breast cancer dataset.

Attributes	Std.error Min/Max	Mean Min/Max	Worst Case Min/Max
Radius (mean of distances from center to points on the perimeter)	0.11/2.87	6.98/28.11	7.93/36.04
Texture (standard deviation of gray-scale values)	0.36/4.89	9.71/39.28	12.02/49.54
Area	6.80/542.20	143.50/2501.00	185.20/4254.00
Smoothness (local variation in radius lengths)	0.00/0.03	0.05/0.16	0.07/0.22
Compactness (perimeter^2/area - 1.0)	0.00/0.14	0.02/0/35	0.03/1.06
Concavity (severity of concave portions of the contour)	0.00/0.40	0.00/0.43	0.00/1.25
Concave points (number of concave portions of the contour)	0.00/0.05	0.00/0.20	0.00/0.29
Symmetry	0.01/0.08	0.11/0.30	0.16/0.66
Fractal dimension (“coastline approximation” - 1)	0.00/0/03	0.05/0.10	0.06/0.21

Results and discussion

The proposed ensemble framework for classifier is shown in Figure 4 and tested on three standard data to establish its robustness. The training phase simulation of BrNdT model has been done with 70% data for training and 30% data for testing. The existing algorithms are also simulated with same standard datasets and their performance metrics are recorded for comparison.

The performance analysis of each model is measured in terms of accuracy and time complexity.

Accuracy is one of the important performance parameters of any classification scheme, since it aids in predicting the correct class and can lead to accurate diagnosis of the patients. A performance comparison of classification accuracy of various classifier schemes is shown in Figure 5. The speed (Computing time) of the classifier also plays an important role. A classifier with smaller accuracy might be preferred over the higher one if shows significant smaller time complexity. Figure 6 shows the comparison of time complexity performance involved in various classifiers.

Other performance parameters such as sensitivity and specificity of the classifier are evaluated from confusion matrix. Confusion matrix consists of two categories of class namely actual and predicted class. The actual classis determined through angiographic method whereas predicted class is simulated through algorithms.

In confusion matrix, four parameters of actual class called True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) are evaluated and listed in Table 6.

Table 6. Confusion matrix.

Predicted Class	Actual Class
	C1	C2
C1	TP	FP
C2	FN	TN

The parameter TP represents the count of instances of Class C1 which are correctly classified as C1 and TN is the count of instances of Class C2 which are correctly classified as C2. Similarly, FN is the count of instances of Class C1 which are falsely classified as C2 and FP represents the count of instances of Class C2 which are falsely classified as C1. The calculated values of Confusion matrix are given in Table 7, Table 8 and Table 9.

Table 7. Confusion matrix for pima indian diabetes dataset.

	TP	FP	TN	FN
BRNN	54	21	194	39
DT	61	38	177	32
LM	54	22	193	39
SVM	58	21	194	35
BrNdT	65	22	193	28

Table 8. Confusion matrix for cpg island dataset.

	TP	FP	TN	FN
BRNN	21	9	120	49
DT	14	4	125	56
LM	5	1	128	65
SVM	0	0	129	70
BrNdT	14	1	128	56

Table 9. Confusion matrix for breast cancer dataset.

	TP	FP	TN	FN
BRNN	55	8	165	0
DT	53	16	157	2
LM	53	5	168	2
SVM	54	5	168	1
BrNdT	54	4	169	1

Based on the parameters used in Confusion Matrix, the performance metric such as sensitivity, specificity, and accuracy are evaluated as given in Table 10 and Table 11.

Table 10. Sensitivity and specificity for various classifiers using different datasets.

Dataset	Pima Indian Diabetes Dataset		CpG Island dataset		Breast Cancer Dataset
Classifier	Sensitivity	Specificity	Sensitivity	Specificity	Sensitivity	Specificity
BRNN	58.1	90.2	30	93	100	95.4
DT	65	82.3	20	96.9	96.4	90.8
LM	58.1	89.8	71	99.2	96.4	97.7
SVM	62.4	90.2	0	100	98.2	97.1
BrNdT	69.9	89.8	20	99.2	98.2	97.7

Table 11. Accuracies and time complexity using various classifiers on different dataset.

	Pima Indian Diabetes		CpG Island Dataset		Breast Cancer
Classifiers	Accuracy(%)	Time(sec)	Accuracy (%)	Time(sec)	Accuracy (%)	Time(sec)
BRNN	80.52	19.52	70.85	45.2	96.49	28.69
DT	77.27	18.39	69.85	24.32	92.11	38.43
LM	80.19	32.37	66.83	26.49	96.93	27.61
SVM	81.82	59.55	64.82	17.16	97.37	21.72
BrNdT	83.77	3.7	71.36	3.54	97.81	4.43

(8)

(9)

(10)

(11)

The results are correlated by computing statistical measures such as accuracy and time complexity listed in Table 11. From the analysis done on the above said datasets, it has been found that SVM has better accuracy than DT, Linear Regression and Bayesian regularized Neural Networks. Now Ensemble classifier is designed by using DT, Linear Regression and Bayesian Regularized Neural Networks and the performance measures of Ensemble Classifier are compared with that of the SVM. It has been found that Ensemble Classifier performs best than SVM.

BrNdT has increased the accuracy as well as reduces time complexity to a great extent. So overall it can be analyzed that Ensemble classifiers perform better than individual classifiers in the binary classification.

Conclusions &future scope

An ensemble based Bayesian regularized Neural network decision Tree (BrNdT) model has been proposed for binary classification. The proposed model is tested on three different standard datasets to validate its robustness. The simulation results show that the proposed scheme not only offers a better accuracy but reduces the time complexity of the model significantly compared to other existing methods. In real life multi-category class problems are more challenging and complex to solve. The work can be extended to design a multi-category class classifier model in future.