On the Influence of Class Noise in Medical Data Classification: Treatment Using Noise Filtering Methods

ABSTRACT

Classification systems play an important role in medical decision support, because they allow automatizing and accelerating the data analysis process. However, their quality is based on that of the training dataset upon which the classification models are built. The labeling process of each training example is usually performed by domain experts or automatic systems. When a wrong assignment of class labels to examples is performed, the training process and, therefore, the classification performance, might be negatively affected. This problem is formally known as class label noise. One of the most used techniques to reduce the harmful consequences of mislabeled objects is noise filtering, which removes noisy examples from the training data. This article analyzes the usefulness of such methods in the context of medical data classification. The experiments carried out on several real-world datasets show the importance of noise filtering when class noise affects the data.

Introduction

Medical data analysis and decision making are acknowledged as important yet difficult tasks and typically are based on years of experience of an expert. In this framework, machine learning techniques have received increased attention from the medical community in the last decade (Kononenko 2001; Pombo, Araújo, and Viana 2014). They allow processing of massive and complex data, providing a fast and effective automatic diagnosis (Azar and Hassanien 2015; Krawczyk and Schaefer 2014). Even though the final decision always lies in hands of physicians, such methods facilitate their work by providing a second opinion, which is especially valuable for physicians who are less experienced or who are experiencing decreased awareness due to fatigue or stress.

According to the specialized literature (Kononenko 2001), three main factors must be considered when designing a new medical decision support system:

• Learning dataset. The first step consists of collecting a set of representative examples that will be the basis for the learning process. These objects must be described by using a set of attributes of high discriminative power and a class label for each example, which is usually provided by a domain expert or an automatic procedure.

• Classification method. The second step uses a classification technique to extract the properties of the learning dataset by constructing a model. This model (called a classifier) must be able to generalize the knowledge embedded in the data to new examples.

• Ease of use. The medical decision support system will be used by physicians who are not necessarily machine learning experts. Therefore, it should ideally be designed without needing the configuration of parameters to each specific problem, and it should be sufficiently interpretable to shed additional light on a patient’s condition.

Even though these three components are important when designing a medical decision system, the first one has a particular relevance because it might somehow influence the quality of the subsequent stages. For this reason, this article focuses on the first step concerning the properties of the training dataset.

As previously commented, a domain expert or an automatic procedure is usually responsible for labeling each training example in the medical dataset. However, this process is commonly subject to errors (Malossini, Blanzieri, and Ng 2006; Wu and Zhu 2008). Errors affecting class labels are formally known in the literature as class label noise or simply class noise (Frénay and Verleysen 2014). Class noise could negatively affect the learning process and, therefore, the building time and classification accuracy of the created models (Sáez, Luengo, and Herrera 2013). For these reasons, two approaches have been proposed to deal with these inconveniences (Zhu and Wu 2004):

1. Algorithm-level approaches (Cohen 1995; Quinlan 1993). These methods adapt existing algorithms to properly handle the noise or be less influenced by its presence.

2. Data-level approaches (Brodley and Friedl 1999; Gamberger et al. 1999). These methods preprocess the data, trying to reduce the impact of noise before building classifiers.

Algorithm-level approaches depend on the concrete adaptation of each classification technique, and therefore, they are not directly extensible to other learning algorithms. Otherwise, data-level approaches are independent of the classifier used, which usually makes this type of technique the most popular choice.

Among data-level approaches, noise filters, which remove noisy examples from the training data, are commonly employed due to their benefits with respect to classification accuracy (Brodley and Friedl 1999; Gamberger, Lavrac, and Dzeroski 1996). However, these techniques do not always provide an improvement in performance because, in many cases, their success depends on the characteristics of the data (Sáez, Luengo, and Herrera 2013).

This article examines the usefulness of several of such noise filtering methods when working with the particular properties of several real-world medical datasets, which are affected by varied quantities of class noise. The elimination of potentially noisy examples helps to build more robust classifiers over the data, which generally leads to a better classification performance because the modeling of the knowledge represented by these difficult examples is avoided (Brodley and Friedl 1999). The set of noisy examples detected could be optionally presented to an expert in order to determine their nature, i.e., if these examples really contain errors or they are exceptions to general classification rules that still contain valuable information. The filtered datasets will be used to create classifiers with three learning methods, each of which has a different behavior against noise: (1) the C4.5 robust algorithm (Quinlan 1993), (2) a Support Vector Machine (SVM; Vapnik 1998), considered accurate but being noise sensitive in some cases, and (3) the nearest neighbor (NN) rule (Mclachlan 2004), which is considered very noise sensitive. We will analyze the performance of these algorithms with and without noise filtering based on a series of experiments that allow us to draw conclusions with regard to the danger of class noise in clinical decision support systems.

The rest of this article is organized as follows. “Medical Data Classification in Presence of Class Label Noise” introduces medical classification with noisy data. “Noise Filtering Methods” details the use of noise filters, paying attention to those considered in this work. In “Experimental Framework,” we describe the experimental framework, and in “Analysis of Results,” we analyze the results obtained. In the final section, we enumerate some concluding remarks.

Medical data classification in presence of class label noise

Noise may be present as errors in the source and input of data affecting the quality of the dataset (Wang, Storey, and Firth 1995). In classification, two types of noise are traditionally distinguished in the literature (Zhu and Wu 2004): attribute noise and class noise.

Attribute noise affects the attribute’s values of examples in a dataset, whereas class noise is produced when the examples are labeled with the wrong classes. Class noise is the most disruptive type of noise to classifier performance because mislabeled examples have a high impact when building classifiers (Sáez et al. 2013, 2014; Zhu and Wu 2004).

In medical data classification, class noise can proceed from several sources (Hickey 1996):

• Human errors. Mistakes during the labeling process, which are more prone to occur when work is with complex data, might occur due to weariness, routine, quick examination of each case, or time pressure. Subjectivity can also produce class noise; for example, when a variability in the labeling by several experts exists (Malossini, Blanzieri, and Ng 2006).

• Machine errors. When a machine is responsible for providing automatic labels, the occurrence of design faults or momentary errors could lead to the presence of erroneous labels.

• Digitalization and archiving errors. When creating a digital record of the examined cases, one might incorrectly input a class by simple mistake. The same situation occurs when using historical recordings.

Class noise can affect the system performance not only producing a decrement of classification accuracy (Sáez et al. 2014), but also affecting the complexity of the classifier built in terms of its size and interpretability (Brodley and Friedl 1999). On this account, many works in the literature focus on its treatment (Brodley and Friedl 1999; Sánchez et al. 2003; Wilson 1972). Two main alternatives have been proposed to deal with noisy data:

• Algorithm-level approaches. These are techniques characterized by being less influenced by noisy data. For example, C4.5 (Quinlan 1993) uses pruning strategies to reduce the chances that the trees are overfitting due to noise (Quinlan 1986).

• Data-level approaches. The most well-known type of methods within this group is that of noise filters (Brodley and Friedl 1999; Khoshgoftaar and Rebours 2007). They identify noisy examples, which can be eliminated from the training data. Figure 1 shows an example of the benefits of learning classifiers in the presence of class noise when filtering techniques are applied.

Figure 1. Dataset with two examples with class noise. Figure 1a shows the exemplary decision boundary created when considering the noisy samples as proper ones. Figure 1b shows a decision boundary after the removal of the noisy examples during the filtering. The decision boundary obtained after the filtering is less complex and leads to a better generalization of these two classes.

Because mislabeled examples can negatively affect the final accuracy of medical decision support systems, there is a high risk in real-world applications because we are often dealing with human health and life. For this reason, the possibility of applying noise filters to the data in order to improve their quality should be always carefully studied. The next section briefly reviews previous works on noise filtering, paying special attention to those filters considered in the experimentation phase of this study.

Noise filtering methods

Noise filters, which remove noisy examples from the dataset, are one of the most used approaches when training data are affected by class noise (Brodley and Friedl 1999). The filtered data can then be used as an input to different classifiers (Khoshgoftaar and Rebours 2007)—hence, the computation time needed to prepare the data is required only once. The elimination of mislabeled examples provides some advantages in classifier performance (Gamberger et al. 1999), in contrast to the removal of instances with attribute noise (Zhu and Wu 2004). This is because examples with errors in some attribute’s values still contain valuable information in other attributes, which can help to build the classifier.

These methods are usually employed with classification methods that are sensitive to noisy data and require data preprocessing to address the problem. The separation of noise detection and learning phase has the advantage of avoiding the usage of noisy instances in the classifier building process (Gamberger, Lavrac, and Dzeroski 2000).

Even though several noise filtering schemes are proposed in the literature (Brodley and Friedl 1999; Gamberger et al. 1999; Sánchez et al. 2003; Wilson 1972), they are usually grouped into different categories according to their characteristics. For a complete review about noise filters, the reader may consult the work of Frénay and Verleysen (2014).

Some filters are based on analyzing the neighborhood among examples (Sánchez et al. 2003; Wilson 1972), using to this end the k-NN classifier (Mclachlan 2004). There are noise filters that use the predictions from an ensemble of classifiers in order to get an improvement in noise detection against considering a specific classifier (Brodley and Friedl 1999). Other types of filtering methods propose an iterative removal of noisy examples (Khoshgoftaar and Rebours 2007) to increase the noise filtering accuracy.

The noise filters considered in this work apply different filtering procedures and are well-known representatives of the field. The parameter setup for all the noise filters is shown in Table 1. All of the parameters are the default ones recommended by the authors of such filters, although we have fixed n = 3 partitions for filters based on ensembles to unify all of them. For noise filters that compute distances between examples, the Heterogeneous Value Difference Metric (HVDM) (Wilson and Martinez 1997), which is valid for both nominal and numerical attributes, has been used. All these noise filters are briefly described as follows:

1. Ensemble Filter (EF) (Brodley and Friedl 1999). This classifies the training data using an n-fold cross-validation with C4.5 (Quinlan 1993), NN (Mclachlan 2004), and LDA (le Cessie and van Houwelingen 1992). Finally, EF removes an example if it is misclassified by more than half of the classifiers.

2. Edited Nearest Neighbor (ENN) (Wilson 1972). This algorithm removes those examples that class does not agree with of the majority of its k = 3 nearest neighbors.

3. Iterative-Partitioning Filter (IPF) (Khoshgoftaar and Rebours 2007). IPF removes noisy examples in multiple iterations. In each one, the data is split into n folds and C4.5 is built over each of these subsets to evaluate all the examples. Then, the examples misclassified by the majority of the classifiers are removed and a new iteration is started.

4. Multiedit (ME) (Devijver 1986). This splits the data into n folds. NN classifies the examples from the part x considering the part (x + 1) mod n as the training set and the misclassified examples are removed. This process is repeated until no examples are eliminated.

5. Nearest Centroid Neighbor Edition (NCNE) (Sánchez et al. 2003). NCNE removes the examples misclassified by the k (k = 3) nearest centroid neighbors rule.

6. Relative Neighborhood Graph (RNG) (Sánchez, Pla, and Ferri 1997). This builds a proximity undirected graph. Each vertex corresponds to an example and the set of edges is composed of those that relate two examples, satisfying a neighborhood relation. RNG discards those objects misclassified by the examples that have a neighborhood relation.

Table 1. Parameter setup for the noise filters.

Noise filter	Acronym	Parameters
ensemble filter	EF	voting scheme: majority, n: 3
edited nearest neighbor	ENN	k-value: 3, distance: HVDM
iterative-partitioning filter	IPF	voting scheme: majority, n: 3
multiedit	ME	k-value: 1, subblocks: 3, distance: HVDM
nearest centroid neighborhood edition	NCNE	k value: 3
relative neighborhood graph	RNG	order of the graph: first, distance: HVDM

Experimental framework

This section presents the details of the experimental framework designed to check how the noise filters described in “Noise Filtering Methods” can alleviate the negative impact of mislabeled examples on the final accuracy. The first subsection here describes the real-world medical datasets used. Then, the following subsection shows how noise is introduced into these datasets. Finally, the methodology followed to analyze the results is described, along with the parameter setup for the classification algorithms.

Datasets

The experimentation considers 12 real-world medical datasets describing different types of applications of clinical decision support systems (including the detection of different types of cancer, diseases in the urinary system, kidney, heart and Parkinson’s disease, among others). Most of them are taken from the UCI repository (Lichman 2013), with exception of the breast cancer (Krawczyk and Filipczuk 2014) and hypertension (Krawczyk and Woźniak 2015) datasets. These datasets are shown in Table 2, where #C refers to the number of classes, #E to the number of examples, and #A to the number of attributes (along with the number of real, integer, and nominal attributes). Examples containing missing values are removed from the datasets before their usage. The consideration of an experimental framework with datasets focused on different medical applications and having different characteristics in terms of the number of classes, examples, and attributes, will enable us to extract more general conclusions from the analysis of the results obtained in the experimentation.

Table 2. Medical datasets considered in the experimentation.

Dataset	#C	#E	#A	Dataset	#C	#E	#A
breast tissue	6	106	9 (9/0/0)	mice protein	8	552	77 (77/0/0)
breast cancer	3	675	84 (84/0/0)	Parkinson’s	2	195	22 (22/0/0)
fertility	2	100	9 (2/0/7)	postoperative	3	87	8 (0/1/7)
hypertension	6	1183	18 (3/7/8)	retinopathy	2	1151	19 (16/0/3)
inflammations	2	120	6 (1/0/5)	statlog	2	270	13 (7/0/6)
kidney disease	2	158	24 (6/5/13)	thoracic surgery	2	470	16 (3/0/13)

A brief description of each dataset is given in the following:

1. Pathological breast tissues (breast tissue). This dataset distinguishes between different types of breast tissues (carcinoma, fibroadenoma, mastopathy, glandular, connective, and adipose) based on physical characteristics measured from the skin.

2. Cytological slide examination (breast cancer). This describes the examination of breast cancer cytology slides taken with fine needle biopsy. Several features describing morphological properties of the nuclei of the cells are used to categorize three types of cancer.

3. Seminal quality (fertility). This consists of medical information (childhood diseases, traumas, surgical interventions, frequency of alcohol consumption, etc.) collected to estimate the seminal quality (normal or altered) of different individuals.

4. Hypertension detection (hypertension). This is a multiclass dataset that deals with the diagnosis of first-order hypertension and five types of secondary-order hypertensions. Features are extracted from a set of medical records and examinations.

5. Inflammations of urinary bladder (inflammations). These data describe potential patients, who are characterized by medical variables (temperature, occurrence of nausea, lumbar pain, and so on). The goal is to predict a possible disease of the urinary system.

6. Chronic kidney disease (kidney disease). This deals with the early detection of kidney failure and it was collected from nearly 2 months in a hospital.

7. Mice protein expression (mice protein). This dataset contains expression levels of 77 proteins measured in the cerebral cortex of eight classes of control and Down syndrome mice exposed to context fear conditioning, a task used to assess associative learning.

8. Voice disorder detection (Parkinson’s). This dataset is composed of several biomedical voice metrics. Each example corresponds to a voice recording from people with or without PD. The aim is to discriminate healthy people from unhealthy people.

9. Postoperative choice (postoperative). This dataset tries to determine where patients in a postoperative recovery area should be sent to next. The attributes correspond to body temperature metrics because hypothermia is a significant concern after surgery.

10. Diabetic retinopathy (retinopathy). This dataset contains attributes extracted from the Messidor image set to predict whether an image contains signs of diabetic retinopathy. The attributes represent a detected lesion, an anatomical part, or an image-level descriptor.

11. Heart disease categorization (statlog). This dataset contains attributes describing the diagnosing of cardiac single proton emission computed tomography (SPECT) images. Each example corresponds to a summarized SPECT image (patient), which is classified into two categories: normal and abnormal.

12. Thoracic surgery survivals (thoracic surgery). These data were collected retrospectively at the Wrocław Thoracic Surgery Centre for patients who underwent major lung resections for primary lung cancer in the years 2007–2011. The task is to predict if a patient will survive for 1 year from the date of the surgery.

Class label noise introduction process

The aforementioned 12 datasets might already contain class noise (Garcia, de Carvalho, and Lorena 2015). Thus, some of the objects were mislabeled by the expert and they will mislead the classifier learning. However, the concrete quantity of such mislabeled examples is a priori unknown, and therefore, we cannot properly extract conclusions based on it.

For this reason, it is necessary to control the amount of class noise in each dataset, introducing the noise in a supervised manner. The uniform class noise scheme (Teng 1999) is used to this end: the x% of the examples are selected and their class labels are changed, considering a random one from the set of classes. The noise levels x = 0%, 5%, 10%, 20%, 30% are considered. Thus, 48 new noisy datasets with class noise are created from the 12 base datasets (making a total of 60 datasets).

In order to create a noisy dataset from the original one, the noise is introduced into the training partitions as follows:

1. A level of noise x% of class noise is introduced into a copy of the full original dataset.

2. Both datasets, the original one and the noisy copy, are partitioned into five equivalent folds with the same examples in each one.

3. The training partitions are built from the noisy copy, whereas the test partitions are formed from examples from the base dataset—the noise-free dataset.

Methodology of analysis

The 60 noisy datasets will be preprocessed with the six noise filters (“Noise Filtering Methods”) resulting in 360 new preprocessed datasets. The effect of these filters will be analyzed, comparing the performance obtained for each dataset with three different classifiers: C4.5 (Quinlan 1993), SVM (Vapnik 1998), and NN (Mclachlan 2004). The accuracy estimation of the classifiers in a dataset is obtained by means of a stratified 5-fold cross-validation, averaging the test accuracy results. The parameter setup for the three classification algorithms used is presented in Table 3.

Table 3. Parameter setup for the classification algorithms.

Classifier	Parameters
C4.5	Confidence: 0.25, minimal instances per leaf: 2, prune after the tree building
SVM	Kernel: polynomial, exponent: 1, C: 1.0, tolerance: 0.001, ε = 10^–12
NN	Distance: HVDM

Statistical comparisons among the results of the 60 datasets considered will also be performed. The results of the Friedman Aligned test (García et al. 2010) and the Finner procedure (Finner 1993) will be computed to check the differences among the three classification algorithms when no noise filters are applied. The same tests will be used to compare all six noise filtering methods, considering each classification method independently, in order to find which are the best filters when working with noisy medical data.

Analysis of results

This section is devoted to the analysis of the results obtained. The first three subsections present the performance results obtained by C4.5, SVM, and NN, using each one of the noise filtering methods. We then show the statistical comparisons performed to find the best classifiers and noise filters for the medical datasets that we are working with. The final subsection presents a possible hypothesis for explaining the findings presented in previous sections, along with the lessons learned after carrying out the experimental study.

Results of C4.5

The experimental results obtained with C4.5, with and without filtering in respect to different noise levels, are given in Table 4. These are briefly summarized hereafter:

Table 4. Accuracies (%) obtained by C4.5 with respect to a given class noise level and noise filtering method.

Noise	0	5	10	20	30	0	5	10	20	30	0	5	10	20	30
Dataset	breast tissue					breast cancer					fertility
None	65.11	59.44	54.72	65.97	56.62	89.63	88.89	82.96	76.44	74.22	88	88	88	89	88
EF	68.83	65.02	66.97	69.87	64.11	90.07	90.22	89.19	88	87.41	88	88	88	88	88
ENN	55.71	51.08	46.19	46.19	48.18	90.07	85.93	86.96	84.44	82.07	88	88	88	85	88
IPF	64.03	60.48	65.11	66.93	64.07	89.78	92	90.52	90.52	84.74	88	88	88	88	88
ME	35.8	36.75	31.99	34.85	33.9	82.37	77.63	78.37	77.93	74.37	88	88	88	88	88
NCNE	62.34	66.97	65.11	65.11	61.3	89.93	88.59	90.37	90.37	84.74	88	88	88	88	88
RNG	55.67	53.72	53.64	54.68	52.86	89.04	88	86.07	84.15	80.74	88	88	88	88	88
Dataset	hypertension					inflammations					Kidney disease
None	75.32	76.58	73.96	72.78	68.47	100	100	99.17	98.33	94.17	97.48	97.48	97.48	89.92	94.94
EF	78.19	78.7	78.7	78.11	79.04	100	100	100	100	96.67	97.48	97.48	98.75	97.48	92.48
ENN	78.11	78.45	77.35	77.01	77.6	100	100	100	100	95	90.5	91.13	82.92	82.28	82.92
IPF	77.09	78.87	78.61	77.34	78.87	100	100	100	100	96.67	98.1	96.85	98.1	96.83	96.85
ME	77.09	77.26	77.09	77.09	77.35	96.67	96.67	96.67	96.67	86.67	89.33	91.05	84.88	81.67	80.42
NCNE	78.02	78.1	78.61	77.35	77.18	100	100	100	100	95	97.48	97.48	97.48	97.48	96.83
RNG	77.18	77.52	77.43	77.51	77.94	100	100	100	98.33	92.5	96.21	96.21	96.21	95.54	88.06
Dataset	Mice protein					Parkinson’s					postoperative
None	89.32	88.78	89.14	77.16	67.93	87.18	82.56	85.64	76.41	76.92	70.13	70.13	72.48	68.95	71.24
EF	87.87	86.05	86.78	84.42	80.42	85.64	85.64	86.15	84.1	84.1	71.24	71.24	71.24	70.13	71.24
ENN	86.42	83.89	86.05	84.24	80.98	82.05	81.54	83.59	84.62	82.56	71.24	71.24	72.42	67.91	71.24
IPF	86.06	83.16	84.43	79.73	84.79	82.05	82.56	84.62	83.08	78.46	71.24	71.24	72.42	70.13	71.24
ME	77.9	77.17	75.91	69.93	73.55	83.08	83.08	81.03	81.54	77.95	71.24	68.89	60.92	71.24	65.36
NCNE	89.32	86.96	88.78	85.7	84.98	83.08	84.1	84.1	84.1	82.56	70.13	70.13	71.31	71.31	71.24
RNG	88.76	85.88	85.87	85.32	77	86.67	80	82.05	82.05	78.97	71.24	71.24	72.42	71.24	71.24
Dataset	retinopathy					statlog					thoracic surgery
None	62.12	64.21	63.42	61.86	60.29	77.78	75.93	77.41	74.07	72.22	85.11	84.89	84.68	84.89	83.83
EF	66.38	66.55	65.77	64.99	65.51	76.3	78.52	75.19	77.04	76.3	85.11	85.11	85.11	85.11	85.11
ENN	63.51	64.9	62.99	64.46	63.25	74.81	73.33	72.22	73.7	71.11	85.11	85.11	84.47	85.11	85.11
IPF	63.34	66.12	64.29	64.38	61.77	78.89	79.63	79.63	78.15	75.19	85.11	85.11	85.11	84.26	85.11
ME	62.12	63.51	62.38	64.12	63.86	67.04	66.67	68.52	67.41	68.89	85.11	85.11	84.89	85.11	83.4
NCNE	66.29	64.46	66.2	62.3	62.73	78.89	78.15	80.37	79.26	76.67	85.11	85.11	84.89	85.11	83.19
RNG	64.9	61.86	63.42	63.86	64.55	71.48	71.11	71.85	72.59	68.89	85.11	85.11	85.11	85.11	85.11

1. C4.5 without filtering:

• For datasets without additional noise, C4.5 obtains a satisfactory accuracy for most of the medical datasets (9 out of 12 datasets have a performance higher than 75%).

• However, in the presence of noise, a quick deterioration of the classification accuracy is observed. For example, with only 5% of noise level, breast tissue has a 5.67% of accuracy drop and Parkinson’s a 4.62% drop.

• For higher noise levels, C4.5 becomes highly inaccurate (mice protein has a 21.39% of accuracy drop and breast cancer a 15.41%).

2. C4.5 with filtering:

• For some of the datasets without additional noise (for example, retinopathy and breast tissue using EF), using some noise filters still leads to small improvements in accuracy. This fact can be explained in that these datasets are already affected by some small degree of class noise on their own and can also add further emphasis on the usage of noise filtering in the process of medical pattern classification.

• There are two cases without additional noise (mice protein and Parkinson’s) in which none of the filters improve the performance of C4.5 without preprocessing.

• When considering the presence of noise in the data, the use of noise filters generally improves the results of not preprocessing (with the exception of the mice protein dataset, in which not preprocessing is the best alternative up to a 10% of noise). It is interesting to note the results in the fertility dataset, in which the performance is usually not altered when considering the introduction of noise. This is because this dataset has a low number of examples (100) and most of them (88%) belong to one of the classes. These aspects imply that the amount of noise introduced is low (because it depends on the total amount of examples) and it probably affects the most common class. Thus, the occurrence of these few, and possibly isolated, noisy examples can keep unaltered the learning of this class, and therefore, the classifier performance is usually slightly affected.

• Comparing the performance of the different noise filters, we may observe that the best approaches are, in general, EF, NCNE, and IPF, because they obtain the best results in many of the datasets and noise levels. One should note that the differences among these filters are, in most cases, relatively small.

Results of SVM

The performance results of SVM, which are presented in Table 5, are outlined following:

Table 5. Accuracies (%) obtained by SVM with respect to a given class noise level and noise filtering method.

Noise	0	5	10	20	30	0	5	10	20	30	0	5	10	20	30
Dataset	breast tissue					breast cancer					fertility
None	60.35	61.3	57.58	57.58	51.9	97.04	97.19	96	96.15	94.81	88	88	88	83	84
EF	62.25	61.21	57.53	51.86	52.86	96.59	96.74	96.59	97.04	95.7	88	88	88	85	87
ENN	43.38	45.28	42.42	42.34	41.47	95.56	96	94.67	95.26	94.81	88	88	87	87	84
IPF	62.21	57.49	60.3	55.63	55.67	96.44	96.74	96.3	96.74	95.26	88	88	88	88	88
ME	35.76	35.76	31.95	35.76	34.85	94.22	94.22	94.52	93.93	93.78	88	88	88	88	88
NCNE	61.3	61.26	60.39	52.86	53.81	96.74	96.44	96.59	96.74	95.7	87	88	87	86	86
RNG	42.42	41.47	40.52	44.33	45.15	96.15	96.59	95.41	95.56	95.7	88	88	87	86	86
Dataset	hypertension					inflammations					kidney disease
None	79.12	78.53	78.28	78.11	77.77	100	100	100	100	100	99.38	100	97.46	95.54	97.48
EF	78.61	79.12	78.95	79.12	77.35	100	100	100	100	100	99.38	99.38	99.38	98.08	96.83
ENN	77.43	77.43	77.6	77.52	77.43	100	100	100	100	100	98.1	95.52	91.09	94.29	93.69
IPF	78.7	78.95	79.29	79.37	78.11	100	100	100	100	100	99.38	99.38	99.38	99.35	98.13
ME	77.09	77.09	77.09	77.09	77.35	96.67	96.67	96.67	96.67	87.5	98.1	95.52	90.48	91.73	86.19
NCNE	78.36	78.19	78.28	78.45	77.43	100	100	100	100	100	99.38	99.38	98.73	99.38	99.38
RNG	77.52	77.52	77.52	77.6	77.18	100	100	100	100	100	98.1	97.44	96.19	95.56	92.44
Dataset	mice protein					Parkinson’s					postoperative
None	99.27	98.91	99.46	97.64	96.56	86.67	87.69	84.1	81.03	84.62	70.13	70.13	71.37	66.73	65.49
EF	99.27	98.55	97.64	96.2	95.29	85.64	86.67	86.67	84.1	85.13	71.24	68.95	70.13	67.71	68.89
ENN	98.19	98.01	96.92	96.02	94.57	86.15	85.64	85.13	82.05	84.62	71.24	65.62	65.29	64.58	71.24
IPF	98.73	98.55	97.83	97.47	96.93	85.64	85.13	86.67	82.56	84.1	71.24	71.24	72.42	70.13	71.24
ME	91.14	90.05	90.23	86.23	80.98	84.62	84.1	84.62	83.08	81.54	71.24	71.24	61.96	67.71	65.36
NCNE	99.27	99.09	98.37	97.1	94.75	86.15	86.15	81.03	82.05	86.15	67.78	67.78	69.08	71.18	70.07
RNG	97.83	96.38	96.38	96.19	93.48	86.15	86.15	84.62	82.56	81.54	70.13	67.91	67.91	70.13	71.24
Dataset	retinopathy					statlog					thoracic surgery
None	67.59	67.07	65.94	65.25	64.12	84.44	84.07	82.59	83.7	80	84.47	84.47	84.47	84.68	84.04
EF	64.81	65.16	65.16	63.6	64.46	82.96	82.96	81.85	82.59	81.48	85.11	85.11	85.11	85.11	84.68
ENN	65.59	65.25	65.77	64.55	64.81	83.33	82.59	82.59	81.85	83.33	85.11	85.11	85.11	84.89	85.11
IPF	63.86	65.94	64.2	62.21	62.99	81.11	81.85	81.48	81.48	79.63	85.11	85.11	85.11	84.89	85.11
ME	63.68	63.42	64.12	63.86	62.99	80.74	82.96	81.11	82.22	81.85	85.11	85.11	85.11	85.11	85.11
NCNE	67.07	66.64	65.77	64.81	63.25	82.96	84.07	82.59	81.85	81.85	84.89	85.11	85.11	84.89	83.62
RNG	66.03	65.59	64.81	63.68	64.55	81.48	81.48	81.85	80.74	79.63	85.11	85.11	85.11	84.68	85.11

1. SVM without filtering:

• For datasets without additional noise, SVM obtains good accuracy results for most of the datasets.

• Considering a very low noise level (5%), the accuracy is almost maintained (for example, in hypertension the drop is 0.59%) or even slightly improved.

• For the highest noise level (30%), SVM is also able to maintain better their initial accuracy than C4.5. Even though in some datasets we observe a high drop in performance (breast tissue with a 8.45%) or a low drop (retinopathy, 3.47%), these are usually lower than in the case of C4.5.

2. SVM with filtering:

• Applying noise filters to some of the datasets without induced noise leads to small improvements in accuracy (for example, breast tissue gains a 1.9% by using EF).

• For most of the datasets without noise, the version without filtering is slightly better than versions that consider the use of filters.

• When considering noise in the data, noise filters improve the accuracy of not preprocessing in some noise levels, generally when they are very high. Despite this fact, the results of SVM in the presence of noise are still good in some datasets (mice protein, retinopathy, and statlog) at low and intermediate noise levels.

• The filters obtaining the best results are EF, NCNE, and IPF. There are some datasets in which all the noise filters obtain a good and similar performance, higher than that of SVM without filtering (thoraric surgery and kidney disease). In other datasets (inflammations and fertility), using filtering or not almost does not vary the performance across all the noise levels.

Results of NN

Table 6 shows the experimental results with respect to different noise levels of the NN classifier. They are summarized in the following:

Table 6. Accuracies (%) obtained by NN with respect to a given class noise level and noise filtering method.

Noise	0	5	10	20	30	0	5	10	20	30	0	5	10	20	30
Dataset	breast issue					breast cancer					fertility
None	69.87	66.06	67.06	65.11	58.61	94.67	90.22	86.96	81.63	73.48	77	78	80	79	69
EF	66.15	64.2	71.65	66.1	58.57	94.67	94.37	93.78	94.07	92.74	88	86	88	83	85
ENN	53.77	55.67	55.63	45.28	55.67	91.41	90.37	90.37	88	86.96	88	86	86	84	83
IPF	66.1	64.29	67.92	67.06	61.43	94.52	94.37	94.07	93.93	92.3	88	88	88	88	88
ME	34.81	37.66	31	34.81	33.9	91.26	90.22	90.07	88.44	85.48	88	88	88	88	88
NCNE	64.2	64.2	62.34	63.16	60.48	94.67	94.52	93.93	93.63	91.7	87	88	86	83	78
RNG	57.71	58.57	54.81	61.39	51.9	92.89	92.44	90.67	88.74	87.26	88	88	88	85	85
Dataset	hypertension					inflammations					Kidney disease
None	74.64	70.08	68.64	62.64	57.06	100	98.33	99.17	90	78.33	100	98.1	92.4	89.19	81.67
EF	79.37	79.2	79.04	79.04	79.12	100	100	100	97.5	95.83	100	100	99.38	99.35	97.44
ENN	78.11	77.94	78.11	75.74	76.5	100	100	100	96.67	93.33	98.75	96.21	91.77	93.08	90.58
IPF	78.87	79.21	78.53	78.45	78.95	100	100	100	100	96.67	100	100	99.38	100	98.73
ME	77.09	77.01	77.18	77.01	77.01	96.67	96.67	96.67	96.67	86.67	85.46	89.19	86.75	83.53	82.94
NCNE	78.36	78.44	78.36	78.53	76.16	100	100	100	95.83	96.67	100	100	99.38	100	99.38
RNG	78.28	78.53	78.02	76.92	76.92	100	100	100	96.67	94.17	99.38	97.48	94.96	95.58	87.96
Dataset	mice protein					Parkinson’s					postoperative
None	98.92	95.48	91.13	80.98	72.29	93.85	93.85	84.1	80	77.44	50.52	50.39	48.24	41.44	40.07
EF	97.29	96.56	95.83	95.65	90.4	90.77	91.28	89.23	86.67	84.1	68.89	66.6	67.78	67.78	66.54
ENN	96.02	95.29	95.47	93.3	90.78	89.23	91.28	88.72	86.15	86.67	67.91	70.13	63.14	60.98	65.49
IPF	97.83	98.19	96.93	94.21	95.66	89.74	88.21	88.21	84.1	81.03	71.24	71.24	72.42	70.13	71.24
ME	89.32	86.6	85.51	79.34	73.18	85.64	85.64	86.15	84.62	81.54	71.24	71.24	61.83	68.89	71.24
NCNE	98.37	96.92	97.64	94.02	91.85	92.82	92.82	90.77	87.69	88.21	68.89	68.95	66.73	65.62	65.49
RNG	97.29	96.93	95.3	93.67	91.5	90.77	91.28	89.74	85.64	83.59	68.95	67.78	67.78	64.31	66.54
Dataset	retinopathy					statlog					thoracic surgery
None	60.38	58.91	59.26	58.56	56.91	77.78	77.41	73.7	75.19	66.67	77.02	76.17	73.83	71.91	67.45
EF	62.38	63.77	62.82	63.6	61.86	80.37	80	81.11	80.37	77.41	84.68	82.77	83.83	84.47	81.28
ENN	61.78	61.95	63.17	62.82	60.82	80	78.89	78.52	78.15	75.56	85.11	85.11	84.47	83.19	80.43
IPF	62.73	63.6	64.73	62.12	61.17	80.37	78.52	78.15	78.89	79.63	85.11	85.11	85.11	84.68	85.11
ME	62.99	62.38	63.25	62.9	61.6	77.04	76.67	77.04	75.19	70	84.89	85.11	84.04	85.11	81.28
NCNE	62.9	63.16	62.3	61.43	62.65	83.33	82.59	81.85	81.11	79.63	84.26	84.47	82.55	81.91	77.45
RNG	62.82	63.16	63.6	62.64	61.78	78.52	76.67	76.3	75.93	71.11	82.98	82.77	80.21	80.85	73.19

1. NN without filtering:

• NN obtains a satisfactory accuracy for most of the medical datasets without additional noise (8 out of 12 datasets have a performance higher than 75%).

• The presence of noise makes the performance of NN deteriorate. Some datasets are already affected, considering only a 5% of noise level (breast cancer has a 4.45% accuracy drop and hypertension a 4.56%).

• For higher noise levels, NN becomes more affected by mislabeled examples. For example, considering a 30% noise level, the breast cancer dataset has a 21.19% accuracy drop and the mice protein dataset has a 26.63%.

2. NN with filtering:

• Without additional noise, using noise filters leads to improvements in accuracy in almost all the datasets (with exception of breast tissue, mice protein and Parkinson’s).

• Considering the presence of noise in the data, the usage of filtering strategies improves the results of not preprocessing (with the exception of the Parkinson’s and breast tissue datasets up to 5% of noise). The improvements are highly remarkable in some cases, for example, in postoperative (31.17%) and mice protein (23.37%).

• The best filtering approaches are EF, NCNE, and IPF, which obtain the best results in many of the datasets and noise levels.

Statistical comparative among noise filters

Tables 7–8 present the statistical comparison performed among classification methods (C4.5, SVM, and NN) and among noise filtering methods (considering each classification method independently), respectively. The ranks obtained by the Friedman Aligned procedure (Rank column), which represent the effectiveness associated with each algorithm, and the p-value related to the significance of the differences found by this test (p_FA row) are shown. The p_Fin column shows the adjusted p-value computed by the Finner test.

Table 7. Statistical comparison among classification methods without filtering.

Classifier	Rank	pFin
SVM	51.625	—
C4.5	93	1.400E-05
NN	126.875	0
pFA	2.68E-010

Table 8. Statistical comparison among noise filters and not preprocessing.

C4.5	Rank	pFin	SVM	Rank	pFin	NN	Rank	pFin
None	231.8333	1.000E-06	None	164.7833	7.840E-01	None	318.5167	0
EF	119.875	—	EF	154.4167	—	EF	138.7833	6.122E-01
ENN	252.8417	0	ENN	250.3167	3.000E-05	ENN	240.9917	1.000E-06
IPF	157.45	1.070E-01	IPF	157.6917	8.825E-01	IPF	127.55	—
ME	334.225	0	ME	318.7167	0	ME	278.175	0
NCNE	138.875	3.913E-01	NCNE	162.1	7.911E-01	NCNE	142.025	5.790E-01
RNG	238.4	0	RNG	265.475	2.000E-06	RNG	227.4583	1.000E-05
PFA	1.64E-009		pFin	1.74E-009		pFin	2.285E-009

Table 7 shows that the average rank obtained by SVM is the best. This is followed by C4.5 and NN. The p-value of the Friedman Aligned test shows that the differences found among these methods are significant. Furthermore, the p-values obtained with the Finner test also show that the differences are always significant (lower than 0.1), when comparing SVM with the other two classification methods.

Table 8 focuses on the comparison among noise filters and not preprocessing, considering each classifier independently. The average ranks of Friedman Aligned show that the best methods are EF, IPF, and NCNE (even though their order can change depending on the classifier considered). The p-values p_FA show statistical differences in all the cases. The p-values obtained with the Finner procedure, when comparing the noise filters, are usually very low in all the cases, except when these filters are compared with each other (and when EF is compared with not preprocessing, using SVM).

From the results of these tables, it is possible to conclude that SVM is the best classifier to work with the medical data considered in this study. It behaves well even without using filtering if the noise level is not high enough. The best noise filters are EF, IPF, and NCNE with independence of the classifier used later.

Noise filtering in medical data classification: Lessons learned

This section summarizes the main findings and lessons learned about the usage of noise filters in medical data classification, after carrying out the experimental study presented in the previous section.

1. Classifier choice and noisy medical data. Even though the most robust classifier used in this work was C4.5, SVM generally has obtained the best performance results (either with or without filtering). This fact clearly shows that it is necessary to test all the available classifiers with our medical data before choosing one of them, with independence of our initial thoughts about its robustness to noise. This is in accordance with the no free lunch theorem (Wolpert 2001), which states that there is no single universal classification method.

2. Medical data classification without using noise filters. Generally, all the classifiers tested (C4.5, SVM, and NN) have provided good accuracy when they are used with the medical datasets without additional noise. This fact shows the potential usefulness of machine learning for these types of problems. However, when noise is introduced into the data, their performance accuracy is usually decreased. Therefore, using noise filtering as a preprocessing step should be considered when designing medical pattern classification systems, because they may provide important advantages in some cases.

3. Noise filters and noisy medical data. The application of noise filters generally improves classifier performance when data are severely affected by noise. Filters make classifiers more robust to noise, alleviating the influence of noisy samples. The best filters found in our experiments were EF, IPF, and NCNE. Therefore, they are the recommended methods to be used in medical pattern classification systems.

However, noise filters do not benefit classifier performance with respect to not preprocessing for some of the datasets and noise levels (which are usually low). Higher noise levels in the data allow noise filters to better show their potential. Furthermore, one must note that noise filters do not always provide an improvement in performance results (Sáez, Luengo, and Herrera 2013), and this depends on several aspects, such as the characteristics of the data treated.

Concluding remarks

In this article, we have examined the usefulness of several noise filters when working with real-world medical datasets, which are affected by class noise. Class noise might be a result of human expert error or a result from some erroneous data gathering. Even though the occurrence of these labeling errors could be mitigated by a double-check of the recorded data or the consensus among several experts when labeling the examples, these solutions are usually time consuming and imply a higher cost. Therefore, labeling errors are generally difficult to prevent. Mislabeled examples can strongly influence the learning process, and therefore, their treatment is of high importance when building classifiers.

We have analyzed the performance of three classification algorithms (C4.5, SVM, and NN), with and without noise filtering, considering six different noise filters and 12 medical datasets with different noise levels.

The experimental results have shown that even small levels of class noise can significantly decrease the classification performance. Without filtering, SVM generally has obtained the best performance results. Noise filtering is especially crucial with the highest noise levels. The best filters found in our experiments are EF, IPF, and NCNE.

However, the application of the noise filters does not always improve the performance of not preprocessing. For this reason, it is always recommended to test all the available classifiers and noise filters, because their success usually depends on the concrete characteristics of the medical data treated.

Funding

José A. Sáez was supported by EC under FP7, Coordination and Support Action, Grant Agreement Number 316097, ENGINE European Research Centre of Network Intelligence for Innovation Enhancement (http://engine.pwr.wroc.pl).Bartosz Krawczyk and Michał Woźniak were supported by the Polish National Science Centre under the grant no. DEC-2013/09/B/ST6/02264.

Additional information

Funding

José A. Sáez was supported by EC under FP7, Coordination and Support Action, Grant Agreement Number 316097, ENGINE European Research Centre of Network Intelligence for Innovation Enhancement (http://engine.pwr.wroc.pl).Bartosz Krawczyk and Michał Woźniak were supported by the Polish National Science Centre under the grant no. DEC-2013/09/B/ST6/02264.