Robust Feature Selection Technique Using Rank Aggregation

Figures & data

Table

Algorithm 1. Rank Aggregation
Input:
1) Feature Evaluation Technique (FET) set Q′ and feature set S
Output:
Ranked Feature Set S′
Algorithm:
Steps: Rank Aggregation
2: for j = 0 to |Q′| – 1 do
S″ = S   ⊳ S″ is a temporary variable
4:  Rank S″ using FETj where
add S″ to list L
6: end for
S′ =aggregated feature set from L using Equation (1)

FIGURE 1 Flow diagram of the rank aggregation-based feature selection algorithm. The number of feature evaluation techniques (using different statistical properties of data) is represented by n.

Table

Algorithm 2. K-step Feature Subset Selection
Input:
Feature Set S^′, Dataset D, Classifiers set M
Output:
feature subset of size k
Algorithm:
set , and and with where K is the feature subset,
2: for do
for do
4:   add feature set Kj to S*
learn S* using Mi
6:   Calculate accuracy of Mi and store it in list tempj
end for
8: search tempj for with the highest predictive power for Mi and store in
end for
10: select the MAX() where

Table

Algorithm 3. Robustness Index Calculation
Input
1) Classification models Mi, where , m is the number of classifiers used,
2) Feature set fk generated from p feature selection techniques, where k is number of top features
Output
Robustness Index rp, for each p feature selection technique
Algorithm:
for i = 0 to m – 1 do    ⊳ For each Mi
2:  for j = 0 to p – 1 do
Cp = classification error with fk
4:  end for
Rank each p based on Cp score and save
6: end for
for i = 0 to p – 1 do
8:  Aggregate the ranks across Mi for using Equation (1)
Assign rp = aggregated ranks
10: end for

TABLE 1  Data Sets with Attributes and Instances

Datasets
Dataset	Attributes	Instances
Lung Cancer	57	32
AML	69	927
Mfeat-fourier	77	2000
Arrhythmia	280	452
Madelon	501	2600
Internet-Ads	1559	3279
Leukemia-3c	7130	72
Embryonal Tumor	7130	60

TABLE 2  Results of Paired t-test for Different Datasets

	Lung Cancer		AML			mfeat - fourier			Embryonal Tumo
	Borda	Kemeny		Borda	Kemeny		Borda	Kemeny		Borda	Kemeny
IG	0.0327	0.0401	IG	0.0509	0.0404	IG	0.0212	0.0227	IG	0.0155	0.0197
SU	0.009	0.0074	SU	0.0571	0.0451	SU	0.0227	0.0243	SU	2.7E-04	6.0E-06
CS	0.0046	0.0066	CS	0.0509	0.0404	CS	0.0227	0.0243	CS	0.0225	0.0255
	Madelon		Internet-Ads			Leukemia-3c			Arrhythmia
	Borda	Kemeny		Borda	Kemeny		Borda	Kemeny		Borda	Kemeny
IG	0.0164	0.0164	IG	0.0119	0.0119	IG	0.0015	0.0015	IG	0.0157	0.0283
SU	0.0139	0.0139	SU	0.0119	0.0119	SU	5.3E-04	5.3E-04	SU	0.0049	0.013
CS	0.0164	0.0164	CS	0.0119	0.0119	CS	3.8E-04	3.8E-04	CS	0.0034	0.0084

IG - Information gain, SU - Symmetric Uncertainty, CS - Chi Square.

FIGURE 2 Comparison of classification accuracies in different datasets using Kemeny, Borda, and three single-feature selection techniques.

TABLE 3  Results for F-Measure

	Naïve Bayes Classifier
	Kemeny	Borda	IG	SU	CS
Datasets
AML (25)	0.67	0.67	0.64	0.65	0.64
Mfeat-fourier (9)	0.79	0.79	0.77	0.77	0.77
Arrhythmia (36)	0.67	0.67	0.53	0.56	0.54
Madelon (40)	0.60	0.60	0.53	0.53	0.53
Internet-Ads (60)	0.95	0.95	0.94	0.94	0.94
Leukemia-3c (60)	0.99	0.99	0.81	0.73	0.79
Embryonal Tumor (60)	0.74	0.72	0.62	0.59	0.61
Lung Cancer (8)	0.45	0.45	0.46	0.51	0.40
	J48 Decision Tree
	Kemeny	Borda	IG	SU	CS
Datasets
AML (25)	0.69	0.69	0.64	0.64	0.64
Mfeat-fourier (9)	0.76	0.76	0.73	0.73	0.73
Arrhythmia (36)	0.65	0.65	0.46	0.50	0.48
Madelon (40)	0.75	0.75	0.51	0.53	0.51
Internet-Ads (60)	0.97	0.97	0.95	0.95	0.95
Leukemia-3c (60)	0.89	0.89	0.75	0.51	0.68
Embryonal Tumor (60)	0.68	0.68	0.73	0.72	0.73
Lung Cancer (8)	0.48	0.48	0.47	0.43	0.42
	k-Nearest Neighbor
	Kemeny	Borda	IG	SU	CS
Datasets
AML (25)	0.63	0.63	0.60	0.62	0.60
Mfeat-fourier (9)	0.83	0.83	0.81	0.81	0.81
Arrhythmia (36)	0.62	0.60	0.50	0.48	0.48
Madelon (40)	0.67	0.67	0.50	0.51	0.50
Internet-Ads (60)	0.97	0.97	0.95	0.95	0.95
Leukemia-3c (60)	0.96	0.96	0.63	0.68	0.61
Embryonal Tumor (60)	0.77	0.79	0.64	0.63	0.64
Lung Cancer (8)	0.38	0.38	0.41	0.28	0.52
	AdaBoost
	Kemeny	Borda	IG	SU	CS
Datasets
AML (25)	0.69	0.69	0.63	0.63	0.63
Mfeat-fourier (9)	0.83	0.83	0.81	0.81	0.81
Arrhythmia (36)	0.45	0.45	0.42	0.42	0.42
Madelon (40)	0.61	0.61	0.54	0.54	0.54
Internet-Ads (60)	0.92	0.92	0.91	0.91	0.91
Leukemia-3c (60)	0.90	0.90	0.71	0.59	0.63
Embryonal Tumor (60)	0.73	0.73	0.57	0.56	0.57
Lung Cancer (8)	0.47	0.47	0.47	0.48	0.47
Datasets
AML (25)	0.69	0.69	0.63	0.64	0.63
Mfeat-fourier (9)	0.79	0.79	0.78	0.78	0.78
Arrhythmia (36)	0.71	0.71	0.54	0.55	0.53
Madelon (40)	0.79	0.79	0.54	0.54	0.54
Internet-Ads (60)	0.96	0.96	0.95	0.95	0.95
Leukemia-3c (60)	0.93	0.93	0.73	0.70	0.69
Embryonal Tumor (60)	0.76	0.76	0.62	0.57	0.59
Lung Cancer (8)	0.44	0.44	0.35	0.35	0.33

TABLE 4  Classification Algorithms Used

Classifiers	Settings
Naive Bayes	estimator classes
J48	pruned C4.5 decision tree
KNN	k=3; brute force search algorithm; Euclidean distance function
AdaboostM1	base classifier: Decision Stump,10 boost iterations, % of weight mass for training was 100
Bagging	weak classifier: fast decision tree learner, bag size as 100%, 10 bagging iterations

TABLE 5  Robustness Index Comparison

	Feature Selection Techniques
	Kemeny	Borda	InfoGain	SymmetricU	Chi Square
Datasets
AML (25)	1	1	3	2	3
Lung Cancer (8)	2	1	3	3	4
Arrhythmia (36)	1	2	4	5	3
mfeat-fourier (9)	2	1	4	3	3
Madelon (40)	1	1	2	3	2
Internet-Ads (60)	1	1	2	2	2
Leukemia-3c (60)	1	1	2	3	3
Embyonal Tumor (60)	2	1	3	4	5