An ensemble machine learning approach for classification tasks using feature generation

Abstract

Although machine learning classifiers have been successfully used in the medical and engineering fields, there is still room for improving the predictive accuracy of model classification. The higher the accuracy of the classifier, the better suggestions can be provided for the decision makers. Therefore, in this study, we propose an ensemble machine learning approach, called Feature generation-based Ensemble Support Vector Machine (FESVM), for classification tasks. We first apply the feature selection technique to select the most related features. Next, we introduce an ensemble strategy to aggregate multiple base estimators for the final prediction using the meta-classifier SVM. During this stage, we use the classification probabilities obtained from the base classifier to generate new features. After that, the generated features are added to the original data set to form a new data set. Finally, this new data set is utilised to train the meta-classifier SVM to obtain the final classification results. For example, for a binary classification task, each base classifier has two probabilities (p for one class and 1−p for the other class). In this case, two new features are generated from the combination of probabilities based on these base classifiers. One is the sum of p as new feature 1, and the other is the sum of 1−p as new feature 2. These two new features are then added to the original data set to form the new data set. In the same way, our feature generation method can be easily extended for a multi-class task for generating new features, where the number of features depends on the number of classes. Those generated features from the base estimators (first layer) are added to the original data set to form a new data set. This new data set is used as the input to the second layer (meta-classifier) to obtain the final model. Experiments based on the 20 data sets show that our proposed model FESVM has the best performance compared to the other machine learning classifiers under comparison. In addition, our FESVM has better performance than the original stacking method in the multi-class classification tasks. Statistical results based on the Wilcoxon–Holm method also confirms that our FESVM can significantly outperform the other models. These indicate that our FESVM can be a useful tool for classification tasks, especially multi-classification tasks.

Keywords:

• An ensemble machine learning approach
• feature generation
• support vector machine
• machine learning

1. Introduction

With the rapid development of information technology, a large amount of raw data has been accumulated from various industries of national economy and social life. How to efficiently and accurately extract the useful information from these data with different format has become the key to promote the in-depth development of industry business and achieve intelligent enhancement. In addition, with the growth of computing power and optimissation of algorithms, artificial intelligence technologies represented by machine learning, deep learning, etc., have been developed tremendously in data processing and analysis. They have also been widely used in many fields of social and economic life. The classification tasks are commonly existing in our daily life and machine learning models are widely used to implement inductive classification tasks for the real-world problems. The target of a classification task is to determine a sample belongs to which class. For examples, Gadde et al. (2021) used email titles and contents to distinguish whether an email is spam or not. Their results showed that the model they proposed outperforms previous models in spam detection with an accuracy of 98.5$\mathrm{\%}$. Gaikwad et al. (2021) utilised magnetic resonance imaging scan results to quickly and accurately distinguish whether a tumour is malignant or benign. Machine learning classification is also widely used in other many real-world applications (Chen & Qian, 2021; Gao et al., 2022; Li et al., 2019). The more accurate the classification is, the more valuable the results can be obtained, which is important to improve the classification performance.

Based on the structural risk minimisation principle and statistical learning theory, the Support Vector Machine (SVM) (Vapnik, 1995) has shown high generalisation performance. It is able to effectively solve traditional challenges in machine learning, such as dimensional catastrophe and overfitting. It has outstanding advantages in solving small-scale, nonlinear and high-dimensional pattern recognition problems. It has been successfully applied to solve probability density estimation (Vapnik, 2000), automatic text classification (Ukey & Alvi, 2012), handwritten digit recognition (Katiyar & Mehfuz, 2015), face detection (Jee et al., 2004), etc. Therefore, further research to improve the generalisation ability and practical applicability of SVM is of great importance.

In addition, the ensemble learning (Zhou, 2021) provides a good solution to improve the generalisation ability of classification systems. Its basic idea is to combine multiple classifiers to learn the same problem, and then combine the learning results of each classifier according to a certain strategy. Ensemble learning improves the generalisation ability and reliability by exploiting the diversity of classifiers. With the diversity of the base model (classifier), the ensemble model (classifier) is less prone to overfitting (Fan et al., 2023). In this study, we propose an ensemble model by combining the ensemble learning and SVM. Specifically, multiple classification models with high variability are selected in the first layer and the SVM is used as the last layer to decide the final result. By doing so, a good ensemble learning model can be achieved and the generalisation ability can be further improved. Therefore, our proposed ensemble model not only has the theoretical significance, but also has practical applications in many fields (Liu et al., 2021; Oloso et al., 2018; Sun & Fu, 2022).

2. Related work

The ensemble learning approach was proposed by Hansen and Salamon (1990) in 1990. It provides a good solution to improve the generalisation ability of classification systems. As one of the important research directions in the field of machine learning (Dietterich, 1997), its basic idea is to combine multiple classifiers to learn from the same issue, and synthesise the learning results of each classifier based on certain strategies. The main combination strategies are averaging method, voting method, and learning method. For the learning method, the most representative method is stacking (Breiman, 1996; Wolpert, 1992). Instead of using simple logical processing (e.g. voting) on the results of the weak classifiers, we apply the combination strategy of stacking (K. Wang et al., 2020). That is, the learning results of several weak classifiers are used as the input to another learner to retrain this classifier and get the final result. In this case, the weak classifier is called the base classifier (estimator), and the classifier used for combination is called the meta-classifier. Specifically, for a data set, each base classifier is used to obtain its classification results, and the classification results are combined in some way to form an input of next layer, which are fed into the meta-classifier. Finally, the meta-classifier is utilised to predict the final result. In this case, ensemble learning improves the generalisation ability and reliability by using the difference of classifiers and is less prone to overfitting.

Faiz (2021) proposed an ensemble SVM-based model for Android malware detection. Their results proved that this ensemble model could effectively detect malware and slightly outperformed the individual SVM model. Das et al. (2021) used an ensembled SVM for in EEG-based person identification. Their experimental results demonstrated that the model not only improved the identification accuracy, but also ensured its robustness and reliability. Mounica et al. (2019) used the ensemble SVM for learning disability detection in school-age children. Their study showed that the proposed multilayer ensemble model had better accuracy in detecting the presence of learning disability problems in children compared to individual algorithms. Naqi et al. (2018) used the ensemble SVM model for the detection of pulmonary nodules where experiments showed that the ensemble SVM method not only improved the sensitivity but also reduced other workloads. The ensemble algorithms have also been applied in other fields with good results (Chatzimparmpas et al., 2021; Nekouei & Sartoli, 2019; Zhang & Li, 2021).

Besides the ensemble strategies, the feature selection techniques are also important for improving a model's prediction performance. For example, Sohail et al. (2022) selected most related features for intrusion detection in smart homes to improve the model's efficiency and accuracy. To identify the digital capabilities in university courses, Fan and Chiong (2022) extracted the features from the text and applied the term frequency-inverse document (TF-IDF) technique for feature processing. They found that it is essential to include appropriate number of features in the prediction models. Chiong et al. (2022) utilised the sentiment analysis technique to extract the most important features for stock market prediction. Their results showed that even with a few important features for traditional machine learning model training, their results could outperform the deep learning models with thousands of features used from the TF-IDF. In addition, feature generation is another way to improve a model's performance. J. Wang et al. (2022) proposed a high-quality proposal feature generation pedestrian detection algorithm to improve detection performance. Experiments demonstrate that their method could achieve 5.9$\mathrm{\%}$ AP and 1.5$\mathrm{\%}$ MR-2 improvement compared to the baseline in crowded pedestrian detection. Guo (2021) proposed a method of scene classification prototype calibration for remote sensing images with few shots based on feature generation model. The proposed model is able to improve the representativeness of each category's features. Their results show that the proposed method can deliver excellent classification accuracy for scene classification.

Although the proper ensemble strategy, feature selection technique, or feature generation approach has been used to improve the model's classification performance, to the best of our knowledge, there is no research that combines these three techniques in a model. Thus, in this paper, we propose a novel approach for classification tasks by design appropriate ensemble strategy, feature selection and feature generation techniques.

3. An ensemble machine learning approach for classification tasks

Based on the above research work, we propose an ensemble machine learning approach for classification tasks using feature generation in order to further improve the prediction accuracy. In this method, there are three modules. The first module is the classifier integration module, in which an integrated model is built on the basis of the stacking ensemble model. The second module is for model selection, including the selection of base classifier and meta-classifier. The third module is the data feature processing module, which performs feature selection and new feature generation.

3.1. Stacking ensemble model

The original idea of stacking is to train multiple base classifiers using an initial training set, and each of these base classifiers generates a new training set to train the meta-classifier (Woods et al., 1997). Figure 1 shows the schematic diagram of the stacking algorithm. The first layer of the stacking algorithm has different types of base classifiers, and the correlation between different base classifier algorithms is low. The meta-classifier is generally a multiple linear regression (MLR) algorithm. The training samples are generally used by cross-validation methods. As shown in Figure 2, the data set is first divided into a training set and a test set with 70% samples used for training and the remaining samples for testing. The training set is used for hyper-parameter tuning based on the 5-fold cross-validation. Specifically, for the 5-fold cross validation, 1/5 of the samples are selected as the test set and the remaining samples are used as the training set. The training set is cross-validated where the base classifier is trained and predicted. The prediction result of each base classifier is used as a feature of the meta-classifier training data, and the label of the new sample remains the same as the original one. After training all the base classifiers, a new data set is generated for the training of our meta-classifier. Similarly, the test set serves directly as input to each trained base classifier to get the prediction, and the average of the prediction results is used as a feature of the new sample, which is then predicted using the meta-classifier. Note that the reasons for using 5-fold cross validation are that (1) the model performance is closely related to the training samples used. If we use the randomly split samples for training, we cannot achieve the best hyper-parameters for model testing; (2) as we run the experiments for 5 times, we can avoid some randomness and compare the results with other models with fairness. As can we can see from Figure 1, the original data set is completely discarded in this process. The final result of the classification depends only on the classification of the base classifiers. In this study, we draw on the idea of original stacking ensemble learning. We make use of the information generated by the base classifier while retaining some of the information from the original data set.

Figure 1. Framework of our stacking model.

Figure 2. The first layer of a single model using 5-fold cross-validation.

3.2. Selection of base classifier and meta-classifier

The stacking integration method is a method that combines theoretical knowledge of statistics with machine learning algorithms to obtain better prediction performance by combining different algorithms together and exploiting the advantages of each algorithm. For the building of the stacking integration model, the first step is to determine the base classifiers. In this stage, the selection of base classifiers needs to be considered in terms of accuracy and diversity. In this paper, models with excellent prediction performance are selected. In order to obtain the best ensemble model, the correlation between the base classifiers also needs to be considered. Different algorithms are used as base classifiers. By doing so, our proposed model is able to explore the intrinsic connections existing between data from different spatial and structural perspectives, and then combine their respective algorithmic principles to build different prediction models. Therefore, choosing algorithms with less correlation can take the advantages of different machine algorithms and achieve the purpose of complementarity.

3.2.1. Random forests (RF)

The RF (Breiman, 2001) is a technique for modelling prediction and behaviour analysis. It is an ensemble method based on bagging, which is built on decision trees. An RF is composed of multiple decision trees, and there is no correlation between different decision trees. When we perform the classification task, each decision tree evaluates the input and classifies them separately. As shown in Figure 3, each decision tree gets a classification result of its own, and the RF takes the classification result which has the highest number of results among all decision trees as the final result. Each tree in the classification takes input from the samples in the initial data set. Then features are randomly selected for growing the tree at each node. Each tree in the forest should not be pruned until the prediction is reached decisively at the end of the exercise. In this way, the RF enables any classifier with weak associations to create a stronger classifier. It is relatively fast to train, easy to make into a parallel method, simple to implement, and easy to balance the error for unbalanced data sets. However, its classification performance is not good for small sample data and low-dimensional data (with a small amount of features) because of the overfitting problem.

Figure 3. The framework of RF.

3.2.2. Logistic regression (LR)

The LR (Cramer, 2003) belongs to the generalised linear model, therefore it has a lot in common with multiple linear regression analysis. They have essentially the same form with $w^{\prime }x+b$, where w and b are the parameters to be sought. The difference lies in their different dependent variables, where multiple linear regression takes $w^{\prime }x+b$ as the dependent variable directly $y=w^{\prime }x+b$, while LR takes $w^{\prime }x+b$ as the dependent variable through the function $S(w^{\prime }x+b)$, which corresponds to a hidden state p, $p=S(w^{\prime }x+b)$. Then the value of the dependent variable is determined according to the magnitude of p versus 1−p. Generally, S is a Sigmod function, which maps a real number to the interval $(0,1)$, LR can be dichotomous or multi-categorical in its dependent variable. The essence of LR is based on the assumption that the data obey this distribution, and then using great likelihood estimation to do the estimation of the parameters. The idea of LR is to fit a decision boundary (not limited to linear, but also polynomial). Next, it establishes a probability link between this boundary and the classification for obtaining the probability in the dichotomous case. LR has a simple form and is faster to train, but the accuracy is not insufficient.

3.2.3. Gaussian Naive Bayes (GNB)

The Bayesian classification (Cheeseman & Stutz, 1996) algorithm is a statistical classification method, which is a class of algorithms that use knowledge of probability statistics for classification. In many occasions, the Naive Bayes (NB) classification algorithm is comparable to decision trees and neural network classification algorithms. It can be applied to large databases with the high classification accuracy and high speed. Gaussian Naive Bayes (GNB) is a classification technique for machine learning based on probabilistic methods and Gaussian distributions. The GNB assumes that the input eigenvalues are continuous and follow a Gaussian distribution, and that each parameter (also called a feature or predictor variable) has the independent ability to predict the output variable. The predictive combination of all parameters is the final prediction, which returns the probability of the dependent variable being classified into each group. The final classification with highest probability is assigned to the group (class).

3.2.4. Multilayer perceptron (MLP)

The MLP (Kothari & Oh, 1993) is a generalisation of the Perceptron Learning Algorithm (PLA), which is described by the structure of neurons as a single. The neural network of the perceptron is represented as follows: as shown in Figure 4, the PLA is a linear binary classifier, but it cannot classify the nonlinear data effectively. Therefore, there is a deepening of the network hierarchy. Theoretically, a multilayer network can model any complex function. The MLP does not specify the number of hidden layers, so the number of layers can be chosen according to the respective needs. The structural model of MLP neural network is shown below. There is only one hidden layer involved in Figure 5, with only three variables and one bias in the input, and three neurons in the output layer. The neuron model is ensemble compared to the neuron model in the perceptron algorithm.

Figure 4. Neuronal structure diagram of the perceptual machine.

Figure 5. Structure of a multilayer neural network with only one hidden layer.

3.2.5. K-nearest neighbour (KNN)

The KNN method originally proposed by Cover and Hart (1967) in 1967, is a theoretically mature method and one of the simplest machine learning algorithms. The idea of KNN is very simple and intuitive: given a sample, if the K most similar (i.e. the nearest neighbour) samples in the feature space belonging to a certain class, then this sample also belongs to that class. As shown in Figure 6, the selection of different of K directly affect the classification result of that sample. This method decides the class to which the sample to be classified belongs based on the class of the nearest sample or samples only in the class determination decision. The KNN method is simple, easy to understand, easy to implement, and does not require estimation of parameters.

Figure 6. When K takes values of 3 and 5, the samples belong to different classifications.

3.2.6. SVM

The SVM is a kernel-based method with high computational power for classification and regression tasks. Compared to other machine learning methods, the SVM has better generalisation capabilities (Fan & Gou, 2023). It has a solid theoretical foundation and its learning strategy is interval maximisation, which can be formalised as a problem of solving convex quadratic programming. In this case, it is also equivalent to the problem of minimising a regularised hinge loss function. The learning algorithm of SVM is an optimisation algorithm for solving convex quadratic programming. The basic idea of SVM learning is to solve for the separation that correctly partitions the training data set and maximises the geometric interval hyperplane. The separation hyperplane is shown in Figure 7. There are an infinite number of such hyperplanes for linearly divisible data sets (i.e. perceptron), but the geometrically spaced separation hyperplane is the only one. SVM can also classify nonlinear data by moving the data to a higher dimensional space through a method called the kernel trick. Often, data that cannot be linearly separated in the original input space can be separated in a high-dimensional feature space. The SVM can obtain accurate classification results with a small amount of samples and provides more accurate results than other algorithms in many applications.

Figure 7. Schematic diagram for finding the maximum interval hyperplane.

3.3. Feature processing

In this section, we conduct the data feature processing in order to improve the prediction performance of our proposed model. There are two main steps: the first step is to eliminate unimportant features through feature selection. The second step is to use base classifier to generate new data features. On the one hand, the data dimension is reduced to improve the computing speed; on the other hand, after the data dimension is reduced, the feature importance of the new features created in Step 2 can be improved in the data set.

3.3.1. Feature selection

Feature selection is an important technique in feature engineering, whose goal is to find the optimal subset of features. Its purpose is to eliminate irrelevant or redundant features, thereby reducing the number of features, improving model accuracy, and reducing runtime. On the other hand, the selection of truly relevant features simplifies the model and assists in understanding the process of data generation. Feature selection, as an important data preprocessing process, has two main advantageous functions: (1) it can reduce the number of features and reduce the dimensionality, which makes the model more generalisable and reduces overfitting; (2) it can enhance the understanding between features and feature values.

In this study, a feature selection method using model-based feature ranking is applied, the idea of this method is to use a machine learning algorithm directly to build a prediction model for each individual feature and response variable. Tree-based methods are easier to use because they model nonlinear relationships better and do not require much debugging. The overfitting problem can be avoided by using cross-validation methods while ensuring that the depth of the tree is not too large.

The XGBoost (Friedman, 2001) is an ensemble algorithm based on boosting, and its main idea is to combine hundreds of tree models into one model with high accuracy, and this model generates new trees by iteration. The importance of a feature is the sum of the number of times it appears in all trees. This means that the more an attribute is used in the model to build a decision tree, the more important it is. There are several types of importance in the XGBoost, it can be computed in several different ways. There are three major ways to calculate the importance of a feature: weight, gain and cover. The weight shows the number of times the feature is used to split data. The gain type shows the average gain across all splits where feature was used. The cover shows the sum of the second-order derivatives of the samples segmented to that node. There are also total_gain, total_cover types of importance. The XGBoost model used in this study is constructed using scikit-learn. Under this model, the XGBoost built-in feature importance uses gain as the evaluation method of importance by default. In the situation where gain is used, the feature importance score is obtained. By setting a threshold, the data set is transformed into multiple subsets with the selected features. By separately putting the subsets into the XGBoost for training and prediction, we can make all the features in the subset with the highest accuracy of XGBoost to be the important features we need to select.

3.3.2. Feature generation

Feature generation is the process of constructing new features from the original data, also known as variable selection, attribute selection, or variable subset selection in machine learning or statistics. It is the process of selecting relevant features and forming a subset of features in model construction. Generating new features based on existing features increases the non-literary of the features. Common data transformations are polynomial-based, exponential function-based, and logarithmic function-based. The new features introduced in feature engineering need to be verified that it can indeed improve the prediction accuracy, rather than adding a useless feature to increase the complexity of algorithm operations.

The feature generation method in this study is different from the original stacking framework. In the original stacking framework, only the prediction results of the base classifier (usually label information) are ensemble into the new data set as the input of the meta-classifier, as shown in Figures 1 and 2. In this study, in order to improve the positive impact brought by the base classifiers, the classification probabilities predicted by all base classifiers. Taking binary classification tasks as an example, the probability of classification as 0 and the probability of classification as 1, are superimposed to obtain the sum of 0 and 1 probabilities of classification of data by all base classifiers. These two pieces of information are added to the data set as constructed new features before putting them into the model for learning and prediction. The same with multi-class classification tasks as shown in Figure 8.

Figure 8. Schematic diagram of the feature generation process.

3.4. Detailed process of our proposed approach for classification tasks

Considering the medical and engineering data collected from the internet may not always clean or some features might not be relevant to the problems, we propose a novel ensemble model based on the techniques of ensemble learning, feature selection, and feature generation. The feature selection technique is first used to select the most important features to improve the prediction accuracy. Next, the feature generation approach is applied to improve the representativeness of each category's features. Finally, the stacking strategy is utilised to achieve the final prediction results. As our proposed FESVM records every possible probability, which remains the probability of positive correct classification, this makes the model somewhat stable.

The complete process of our proposed ensemble model, the FESVM, is shown in Figure 9. As can be observed from the figure, we perform three main modules for a classification data set. The first module is data preprocessing. In this module, we take the input $Datase{t}_0$ for data normalisation and select important features using the XGBoost, and finally, we get the $Datase{t}_1$ which only contains important features. The second module is the generation of new features. In this module, we use multiple heterogeneous machine learning models to generate prediction probabilities for the corresponding classifications, and add the obtained sum of probabilities to $Datase{t}_1$ to obtain $Datase{t}_2$. In the third module, we perform the final training and prediction. The latest $Datase{t}_2$ is injected into the SVM to obtain the final prediction results. The default parameters of each algorithm in scikit-learn are used throughout the model. Our proposed ensemble machine learning approach combines these three modules for classification tasks using feature generation.

Figure 9. Schematic diagram of the experimental model frame structure.

For the classification tasks, the idea of ensemble learning is to correct the incorrect prediction of a weak classifier using the correct prediction of other weak classifiers. It is clear that the heterogeneity of classifiers (i.e. individual variability) facilitates the classification performance. As shown in Figure 9, the RF, LR, MLP, GNB, and KNN are chosen as the base classifiers while the SVM is selected as the meta-classifier in this study. After the data normalisation, the K-fold cross-validation is applied with feature selection and feature generation used, followed by the meta-classifier training to get the final model.

4. Experimental results and analyses

4.1. Data description and organisation

The 20 data sets used for our experiments were downloaded from the UCI and Kaggle for binary classification and multi-classification problems, covering several domains and different number of features and samples, as shown in Table 1 for detailed information. The smallest data set has 116 samples while the biggest data set has 65,532 samples. The number of features ranges from 3 to 60.

Table 1. Data set information where #Instances is the number of instances and #Features is the number of features.

Data set	#Instances	# Features	Type
BanknoteA (Lohweg, 2012)	1372	4	Binary
BreastCC (Patrício et al., 2018)	116	9	Binary
ChessKR&KP (Shapiro, 1989)	3196	36	Binary
ConBSMR (Sejnowski & Rosenberg, 1987)	208	60	Binary
DiabetesBC (Madan, 2022)	768	8	Binary
HeartAA&PD (Rahman, 2022)	303	13	Binary
RoomO (Mehra, 2022b)	2665	5	Binary
SPECTFHeart (Krzysztof J. Cios, 2001)	267	44	Binary
Statlog (Heart) (Dua & Graff, 2017)	270	13	Binary
VertebralC (de Alencar Barreto & da Rocha Neto, 2011)	310	6	Binary
BMI (H, 2022)	500	3	Multi-Class
3WineC (Mehra, 2022a)	178	13	Multi-Class
Carevalu (Bohanec, 1997)	1728	6	Multi-Class
DryBean (Koklu & Ozkan, 2020)	13611	16	Multi-Class
ForestTM (Johnson et al., 2012)	521	27	Multi-Class
InternetFD (Ertam, 2019)	65532	11	Multi-Class
Iris (Fisher, 1988)	150	4	Multi-Class
TurkishME (Er & Aydilek, 2019)	400	50	Multi-Class
UltrasonicFD (Gyamfi et al., 2018)	180	43	Multi-Class
WebsiteP (Abdelhamid et al., 2014)	1353	9	Multi-Class

Table 2 shows the relationship between the original features, the important features, and the newly generated features in the data set. The most important features are selected to ensure the highest accuracy of the model. The ‘Important features’ in the table is the number of important features left. ‘New Features’ is the number of new features obtained from the base classifier, which is as many as the number of categories in the data set. ‘Proportion of Reservation’ shows the proportion of important features retained in the number of original features. ‘Proportion of New Features’ shows the proportion of new features among all features (i.e. Important features and New features).

Table 2. Data set information after preprocessing.

Data set	Important features	New features	Proportion of reservation	Proportion of new features
BanknoteA	4	2	1.00	0.33
BreastCC	7	2	0.78	0.22
ChessKR&KP	23	2	0.64	0.08
ConBSMR	50	2	0.83	0.04
DiabetesBC	6	2	0.75	0.25
HeartAA&PD	10	2	0.77	0.17
RoomO	5	2	1.00	0.29
SPECTFHeart	11	2	0.25	0.15
Statlog (Heart)	5	2	0.38	0.29
VertebralC	5	2	0.83	0.29
BMI	3	6	1.00	0.67
3WineC	13	3	1.00	0.19
Carevalu	5	4	0.83	0.44
DryBean	15	7	0.94	0.32
ForestTM	21	4	0.78	0.16
InternetFD	9	4	0.82	0.31
Iris	4	3	1.00	0.43
TurkishME	50	4	1.00	0.07
UltrasonicFD	43	4	1.00	0.09
WebsiteP	7	3	0.78	0.30

Our proposed FESVM is proposed to deal with the classification tasks. In the experiments, the evaluation metrics considered were accuracy, precision, recall and F1-score. We pay more attention to the performance results of accuracy. Based on the above-mentioned metrics, we compare the single machine learning model used in this study, the original stacking model, and the FESVM model. In order to make an effective comparison, we do not optimise the parameters for each method and use the default parameters with 10-fold cross validation.

4.2. Experimental results and analyses

Tables 3–5 show the experimental results from the RF, LR, GNB, MLP, KNN, SVM, Stacking, and FESVM based on the 20 data sets. From these tables, it is clear to see that our proposed FESVM algorithm has excellent results. To be specific, our proposed FESVM has the best performance among 16 of the 20 data sets in terms of accuracy; our proposed FESVM has the best performance among 15 of the 20 data sets in terms of precision; and our proposed FESVM has the best performance among 16 of the 20 data sets in terms of F1-score. This is mainly due to the use of the ensemble strategy used in our model. In addition, when compared with the original stacking algorithm, FESVM performs better than Staking, especially on the multi-classification data set. This could be the use of feature selection and generation that could refine the feature information to further improve our FESVM.

Table 3. Experimental results from the RF, LR, GNB, MLP, KNN, SVM, Stacking, and FESVM using the first eight data sets in Table 1.

Data set	BanknoteA				BreastCC
Methods	Accuracy	Precision	Recall	F1	Accuracy	Precision	Recall	F1
RF	0.9920	0.9871	0.9954	0.9912	0.7220	0.7214	0.7250	0.6523
LR	0.9716	0.9509	0.9849	0.9673	0.7167	0.6367	0.8056	0.6847
GNB	0.8448	0.8314	0.8193	0.8229	0.6356	0.5637	0.8917	0.6615
MLP	0.9912	0.9818	0.9982	0.9899	0.6818	0.6455	0.7597	0.6525
KNN	0.9978	0.9953	1.0000	0.9977	0.7167	0.6827	0.7439	0.6711
SVM	0.9993	0.9982	1.0000	0.9991	0.7697	0.7167	0.8347	0.7338
Stacking	0.9709	0.9766	0.9543	0.9653	0.7714	0.7143	0.7143	0.7143
FESVM	0.9993	0.9982	1.0000	0.9991	0.7742	0.7650	0.7653	0.7291
	ChessKR&KP				ConBSMR
RF	0.9934	0.9928	0.9947	0.9937	0.8269	0.8520	0.8200	0.8030
LR	0.9603	0.9657	0.9583	0.9620	0.7643	0.7545	0.7745	0.7427
GNB	0.8432	0.7940	0.9447	0.8626	0.7019	0.6532	0.8306	0.7147
MLP	0.9922	0.9928	0.9922	0.9924	0.7976	0.7936	0.7730	0.7709
KNN	0.9628	0.9491	0.9817	0.9650	0.7981	0.8053	0.7652	0.7689
SVM	0.9931	0.9951	0.9916	0.9933	0.8364	0.8479	0.8394	0.8208
Stacking	0.9833	0.9825	0.9863	0.9844	0.7619	0.8000	0.6667	0.7273
FESVM	0.9934	0.9940	0.9935	0.9937	0.8457	0.8481	0.8326	0.8276
	DiabetesBC				HeartAA&PD
RF	0.7617	0.6448	0.6909	0.6626	0.8216	0.8223	0.8428	0.8274
LR	0.7461	0.6193	0.7146	0.6588	0.8385	0.8296	0.8717	0.8475
GNB	0.7552	0.6348	0.6903	0.6584	0.8155	0.8317	0.8174	0.8195
MLP	0.7604	0.6377	0.7224	0.6727	0.8419	0.8417	0.8686	0.8509
KNN	0.7305	0.5924	0.7229	0.6468	0.8281	0.8388	0.8242	0.8271
SVM	0.7461	0.6138	0.7192	0.6578	0.8219	0.8107	0.8741	0.8359
Stacking	0.8182	0.7681	0.6709	0.7162	0.8022	0.7966	0.8868	0.8393
FESVM	0.7643	0.6370	0.7496	0.6836	0.8513	0.8605	0.8532	0.8546
	RoomO				SPECTFHeart
RF	0.9854	0.9744	0.9853	0.9797	0.8168	0.8891	0.8776	0.8816
LR	0.9779	0.9447	0.9978	0.9704	0.7613	0.9673	0.7233	0.8239
GNB	0.9554	0.8968	0.9927	0.9417	0.7422	0.9652	0.6985	0.8089
MLP	0.9782	0.9465	0.9968	0.9708	0.7160	0.9817	0.6544	0.7827
KNN	0.9850	0.9693	0.9895	0.9793	0.7120	0.9474	0.6748	0.7842
SVM	0.9797	0.9475	1.0000	0.9729	0.7719	0.9341	0.7696	0.8411
Stacking	0.9825	0.9554	1.0000	0.9772	0.8272	0.8354	0.9851	0.9041
FESVM	0.9854	0.9774	0.9822	0.9797	0.8242	0.8943	0.8819	0.8861

Table 4. Experimental results from the RF, LR, GNB, MLP, KNN, SVM, Stacking, and FESVM using the ninth to sixteenth data sets in Table 1.

Data set	Statlog (Heart)				VertebralC
Methods	Accuracy	Precision	Recall	F1	Accuracy	Precision	Recall	F1
RF	0.8296	0.8326	0.8578	0.8406	0.8419	0.7186	0.8056	0.7570
LR	0.8148	0.8429	0.8130	0.8215	0.7677	0.5894	0.8104	0.6789
GNB	0.8148	0.8307	0.8317	0.8262	0.7774	0.5951	0.8631	0.7000
MLP	0.8222	0.8410	0.8230	0.8276	0.8065	0.6369	0.8370	0.7206
KNN	0.7815	0.7992	0.8001	0.7934	0.7677	0.5955	0.7663	0.6683
SVM	0.8296	0.8426	0.8371	0.8369	0.8065	0.6417	0.8375	0.7236
Stacking	0.8519	0.8491	0.9184	0.8824	0.7849	0.7826	0.5455	0.6429
FESVM	0.8444	0.8563	0.8480	0.8498	0.8484	0.7461	0.7583	0.7495
	BMI				3WineC
RF	0.8700	0.8440	0.8347	0.8272	0.9830	0.9811	0.9877	0.9830
LR	0.7740	0.6771	0.7136	0.6710	0.9719	0.9683	0.9777	0.9696
GNB	0.6740	0.6578	0.6427	0.6317	0.9663	0.9669	0.9769	0.9685
MLP	0.8960	0.8412	0.8550	0.8352	0.9719	0.9743	0.9711	0.9692
KNN	0.8680	0.8300	0.8483	0.8185	0.9441	0.9406	0.9580	0.9412
SVM	0.8640	0.8318	0.8384	0.8175	0.9889	0.9889	0.9847	0.9851
Stacking	0.6467	0.5001	0.4642	0.4491	0.9444	0.9380	0.9508	0.9424
FESVM	0.9020	0.8582	0.8606	0.8529	0.9830	0.9836	0.9869	0.9840
Data Set	Carevalu				DryBean
RF	0.9618	0.8825	0.9519	0.9078	0.9242	0.9357	0.9351	0.9352
LR	0.8188	0.6580	0.8600	0.7152	0.9160	0.9293	0.9309	0.9296
GNB	0.6909	0.5446	0.7854	0.5648	0.8982	0.9074	0.9093	0.9078
MLP	0.9508	0.8707	0.9472	0.8992	0.9221	0.9341	0.9353	0.9343
KNN	0.9433	0.8238	0.8607	0.8323	0.9157	0.9274	0.9291	0.9280
SVM	0.9589	0.8801	0.9508	0.9058	0.9244	0.9370	0.9371	0.9367
Stacking	0.8767	0.7320	0.5703	0.6212	0.8935	0.8860	0.9010	0.8919
FESVM	0.9618	0.8825	0.9519	0.9078	0.9251	0.9369	0.9359	0.9362
	ForestTM				InternetFD
RF	0.8814	0.8805	0.8722	0.8677	0.9829	0.7562	0.8998	0.7552
LR	0.8853	0.8823	0.8841	0.8757	0.8814	0.7172	0.7932	0.6757
GNB	0.8528	0.8445	0.8575	0.8431	0.9921	0.7556	0.8033	0.7612
MLP	0.8967	0.8947	0.8829	0.8805	0.9617	0.7506	0.8934	0.7344
KNN	0.8892	0.8754	0.8829	0.8739	0.9879	0.7596	0.8943	0.7678
SVM	0.8967	0.8908	0.8870	0.8821	0.9119	0.7288	0.8008	0.6949
Stacking	0.7962	0.7702	0.7860	0.7759	0.9977	0.7478	0.7486	0.7482
FESVM	0.8987	0.8921	0.8913	0.8850	0.9924	0.7668	0.8966	0.7782

Table 5. Experimental results from the RF, LR, GNB, MLP, KNN, SVM, Stacking, and FESVM using the last four data sets in Table 1.

Data Set	Iris				TurkishME
Methods	Accuracy	Precision	Recall	F1	Accuracy	Precision	Recall	F1
RF	0.9467	0.9476	0.9408	0.9380	0.8000	0.8020	0.8072	0.7966
LR	0.9333	0.9282	0.9260	0.9233	0.8050	0.8109	0.8086	0.7988
GNB	0.9467	0.9429	0.9427	0.9375	0.7450	0.7496	0.7430	0.7286
MLP	0.9267	0.9322	0.9305	0.9219	0.7800	0.7837	0.7866	0.7742
KNN	0.9467	0.9475	0.9383	0.9373	0.6425	0.6521	0.6442	0.6214
SVM	0.9533	0.9512	0.9494	0.9460	0.7900	0.7915	0.7994	0.7852
Stacking	0.9111	0.8867	0.8867	0.8867	0.5917	0.7056	0.5539	0.5398
FESVM	0.9533	0.9512	0.9494	0.9460	0.8150	0.8233	0.8213	0.8120
	UltrasonicFD				WebsiteP
RF	0.8944	0.8985	0.9080	0.8961	0.8885	0.8634	0.8841	0.8690
LR	0.8500	0.8705	0.8502	0.8532	0.7739	0.6840	0.7528	0.6844
GNB	0.8611	0.8892	0.8620	0.8651	0.7694	0.6760	0.7279	0.6721
MLP	0.7333	0.7733	0.7929	0.7606	0.8914	0.8392	0.9044	0.8589
KNN	0.8611	0.8776	0.8678	0.8600	0.8721	0.8155	0.8785	0.8327
SVM	0.7111	0.7641	0.7257	0.7267	0.9039	0.8513	0.9194	0.8727
Stacking	0.4815	0.7088	0.5799	0.4718	0.7857	0.6422	0.6421	0.6370
FESVM	0.9167	0.9237	0.9225	0.9210	0.9040	0.8732	0.9070	0.8860

In specific analysis, the original stacking is severely affected by the base classifiers because it discards the features in the original data and only relies on the classification results provided by the classifiers. Similarly, the original stacking is also affected accordingly when the base classifiers have poor performance, such as ConBSMR in Table 3, BMI in Table 4, and the UltrasonicFD in Table 5. However, our proposed FESVM not only retains the feature information in the original data, but also integrates the feature information provided by the base classifiers, maximising the value of both information. Therefore, FESVM does not suffer from the poor classification results from some base classifiers on these data sets with poor performance, but obtains better results. Although for the DiabetesBC and SPECTFHeart in Table 3 and Statlog (Heart) and InternetFD in Table 4, Stacking performs better than the FESVM, their results are still competitive. This is most likely due to the fact that too many important features are screened out during the feature screening process, causing the FESVM algorithm lose some information while the feature generation technique, to some extent, could alleviate its influence. As our target is to improve the accuracy, our proposed FESVM is able to guarantee its accuracy to be higher than or equal to other base classifiers in most cases. It has the same result as SVM on data sets 1 and 17, and the same result as the RF on data set 13, but performs well on the remaining data sets. This could be the reasons that (1) the SVM is based on the structural risk minimisation principle and statistical learning theory, making it suitable for small data sets. While our proposed FESVM is based on the SVM, which is also applicable in such scenarios; (2) the RF is able to improve the prediction accuracy and stability by integrating multiple base estimators into one model while our proposed FESVM uses the ensemble strategy, which leads to its high performance. These results show that our FESVM can perform well in most cases, indicating it is more universal than the other models under comparison.

In addition, as can be seen from the tables, our FESVM performs better on the multi-classification tasks compared to the binary classification tasks. It also shows superior performance over stacking on these two multi-classification data sets. Each individual algorithm performs more consistently on these two cases, but stacking appears to have better performance for binary classification tasks. The reason is that there are multiple categories, and for each model there are N possible categories (N is the number of categories), then there are N−1 incorrect classifications, then there is a higher probability of stacking errors and more deviations from the correct category. The occurrence of these errors will then make the stacking results deviate from their targets. Our proposed FESVM, on the other hand, records every possible probability, which remains the probability of positive correct classification, making the algorithm somewhat stable.

Furthermore, as we can see from the experimental results of each data set, the results of some data sets show the improvement effect of FESVM is less than 1%, which is a significant improvement. While for some data sets, our FESVM enhancement results are obvious. The reason why we use the average results of 20 data sets to prove the effectiveness of FESVM is that our purpose is to propose a model that can meet the needs of wide applicability while improving the accuracy. Therefore, we aim to prove that our proposed FESVM performs well on most data sets without causing too much extra time. As can be seen from the average of these 20 data sets, the RF model may perform best in a single data set, it performs moderately in other data sets. Among the averaged results, our FESVM has the best performance in terms of all metrics used, which indicates that our FESVM may not be the best on a single data set, but it performs the best on the average of 20 data sets. This confirms that our FESVM is competitive in applicability.

To further verify the performance of our proposed FESVM, the comparison of the average performance results of each model based on the 20 data sets is shown in Figure 10. As we can from the figure, our proposed FESVM algorithm outperforms the other models being compared in terms of all metrics used. In particular, the Accuracy, Precision and F1-score values are much higher than the other algorithms. The average performance of FESVM is not only higher than that of the single-use SVM, but also higher than that of the original stacking, which shows that our FESVM is more flexible and robust.

Figure 10. Comparison of the average performance results of each model based on the 20 data sets.

Although our proposed model has shown better performance than the other models being compared, whether their difference is significant or not is still unknown. Thus, we used the critical difference diagram with Wilcoxon-Holm method (Ismail Fawaz et al., 2019) to show the performance effect of these algorithms in terms of Accuracy and F1-score while the statistical results (p-values) based on the Wilcoxon-Holm method to detect pairwise significance are shown in Tables 6 and 7. Note that in Figures 11 and 12, the value closer to 1 indicates that the algorithm has better performance. The algorithms connected by the thick lines are those with closer performance. As can be seen from the figures, our proposed FESVM has the best performance in terms of Accuracy and F1-Score. The second and third best models are RF and SVM, respectively, which can also be seen in Tables 3–5.

Figure 11. Critical difference diagram of Accuracy based on the 20 data sets.

Figure 12. Critical difference diagram of F1-Score based on the 20 data sets.

Table 6. Statistical test results for the RF, LR, GNB, MLP, KNN, SVM, Stacking, and FESVM in terms of accuracy (p-values less than 0.05 are highlighted in bold).

	LR	GNB	MLP	KNN	SVM	Stacking	FESVM
RF	5.93E−04	1.75E−04	2.06E−02	2.99E−03	0.54	1.11E−02	1.35E−04
LR		1.45E−02	9.66E−02	3.80E−01	8.03E−03	0.91	8.86E−05
GNB			1.11E−02	0.09	6.42E−03	0.30	8.86E−05
MLP				0.15	0.23919	1.08E−01	8.86E−05
KNN					3.04E−02	0.23	8.86E−05
SVM						3.04E−02	4.76E−04
Stacking							2.50E−03

Table 7. Statistical test results for the RF, LR, GNB, MLP, KNN, SVM, Stacking, and FESVM in terms of F1-Score (p-values less than 0.05 are highlighted in bold).

	LOG	GNB	MLP	KNN	SVM	Stacking	FESVM
RF	3.19E−03	2.93E−04	1.69E−02	3.59E−03	0.58	1.52E−02	5.53E−04
LR		1.24E−02	0.08	0.68	4.55E−03	0.17	8.86E−05
GNB			1.37E−02	0.14	4.55E−03	0.94	8.86E−05
MLP				0.11	0.19	4.79E−02	8.86E−05
KNN					1.52E−02	0.09	8.86E−05
SVM						1.87E−02	1.09E−03
Stacking							1.51E−03

Based on the above experimental results, it can be seen that the performance results of individual classifiers on different data sets are poor in terms of performance and generalisation ability depending on the data sets. Ensemble learning can combine the processing results of the main classifiers to obtain better results. Experiments show that FESVM not only performs better than single base classifier on binary classification, but also performs better on multi-classification. In addition, FESVM is far better than stacking, especially in the case of insufficient data sets.

Table 8 shows the computation time (in seconds) based on different input features used for the 20 data sets using our proposed FESVM where t1 is the computation time using all features, t2 is the computation time using generated features, t3 is the computation time using feature selection, and t4 is the computation time using the features from the feature generation and feature selection. Note that for t2, t3, and t4, the computation time of feature processing is included. As shown in the table, the computation time of t2 is smaller than t1. This is expected as less feature used for training, its computation time will be less if the time for feature processing is not much. In addition, the number features used in t3 is less than t2, the computation time for t3 should be less. But it is only true for the BanknoteA data set. This is because the time for feature generation takes more time than the feature selection. Although most of computation time of our proposed FESVM is more than the original features. The efficiency could be improved by running feature selection and feature generation at the same time. By doing so, the computation mainly depends of the maximum computation time of t2 and t3, which is comparable with the computation time based on the original data set.

Table 8. Comparison of computation time based on different input features used for the 20 data sets using our proposed FESVM where t1 is the computation time using all features, t2 is the computation time using generated features, t3 is the computation time using feature selection, and t4 is the computation time using the features from the feature generation and feature selection.

	t1	t2	t3	t4
BanknoteA	21	13	4	17
BreastCC	8	3	5	9
ChessKR&KP	78	40	47	87
ConBSMR	10	5	39	43
DiabetesBC	30	14	10	24
HeartAA&PD	11	5	12	17
RoomO	35	27	23	50
SPECTFHeart	9	6	39	45
Statlog (Heart)	6	5	9	14
VertebralC	7	6	7	12
BMI	18	15	19	34
3WineC	5	4	14	18
Carevalu	79	61	42	103
DryBean	1124	868	1735	2603
ForestTM	15	11	90	101
InternetFD	9568	8470	1938	10,408
Iris	3	4	4	8
TurkishME	10	10	133	143
UltrasonicFD	6	5	62	67
WebsiteP	36	27	38	66

5. Conclusion

In this study, we proposed the FESVM by combining the sum of classification probability in the frame of stacking, which is able to improve the accuracy of the model classification by using the feature generation technique before training the meta-classifier SVM. It is demonstrated experimentally that our proposed FESVM outperformed the other models being compared, including the RF, LR, MLP, GNB, KNN, and SVM, based on the 20 data sets. The FESVM is effective on both binary classification and multi-class classification scenarios. Furthermore, statistical results based on the Wilcoxon–Holm method showed the FESVM could significantly outperform the other models being compared in terms of Accuracy and F1-score. These indicate that our proposed FESVM is effective for both binary and multi-class classification tasks.

With the excellent performance obtained by our proposed FESVM, it could be a useful tool for classification tasks in real-world applications by providing suggestions/information for decision makers to take proper actions. For example, in medical applications, the proposed model could predict if a patient has certain diseases based on the diagnosis data. This is helpful for the doctor or decision maker to know more about the potential outcomes.

Although the results have shown the high performance of our proposed model, it has some limitations in practical applications. These include (1) the ensemble strategy used in the second layer has a great impact on the prediction performance; (2) proper feature processing techniques are also important for classification accuracy and model training efficiency. In future work, we will investigate different combination strategies to generalise the FESVM in other classification tasks. We will explore our strategies for feature processing without expanding the data dimension. For example, the probabilities could be embedded in the original information in some way without generating new features. At the same time, the new model could achieve promising performance with high efficiency.

Disclosure statement

No potential conflict of interest was reported by the author(s).