ParkinsoNET: Estimation of UPDRS Score Using Hubness-Aware Feedforward Neural Networks

ABSTRACT

Parkinson’s disease is a worldwide, frequent, neurodegenerative disorder with increasing incidence. Speech disturbance appears during the progression of the disease. The Unified Parkinson’s Disease Rating Scale (UPDRS) is a gold-standard tool for diagnosis and follow-up of the disease. We aim at estimating the UPDRS score based on biomedical voice recordings. In this article, we study the hubness phenomenon in context of the UPDRS score estimation and propose hubness-aware error correction for feedforward neural networks to increase the accuracy of estimation. We perform experiments on publicly available datasets derived from real-voice data and show that the proposed technique systematically increases the accuracy of various feedforward neural networks.

Introduction

Parkinson’s disease (PD) is one of the most important neurodegenerative disorders and is increasing in incidence. PD affects 7 to 10 million people worldwide. Clinically, PD is characterized by cardinal symptoms: initially unilateral, asymmetrical resting tremor (shaking of the hand), rigidity and bradykinesia (slow movement). In addition to these symptoms (motor disturbances), the disorder is associated with nonmotor, neuropsychiatric symptoms such as cognitive impairment, autonomic dysregulation, and sleep problems (Crosiers et al. 2011).

Total PD-related costs are estimated as $25 billion per year in the United States alone, and “medication costs for an individual person with PD average $2,500 a year, and therapeutic surgery can cost up to $100,000 dollars per patient.”¹ As noted by De Rijk et al. (1997) and De Lau and Breteler (2006), the prevalence increases with the age, and the increasing importance of PD is underlined by the fact that “most economically developed and many developing countries are experiencing marked demographic shifts, with progressively larger proportions of their populations entering old age” (Pringsheim et al. 2014).

The UPDRS is the most commonly used scale in the clinical study of PD (Ramaker et al. 2002). Roughly speaking, the UPDRS score of a particular patient describes the severity of the disease in the case of that patient (see the references in the following subsection for more details on UPDRS). The UPDRS score might change over time, indicating the success of treatment or the progression of the disease. Ideally, the UPDRS score would be measured regularly and relatively often in order to provide medical doctors, the patient, and his/her relatives with detailed information about the progression of the disease and to contribute to the patient’s awareness of the disease, which is one of the most relevant factors influencing the efficiency of the treatment. However, because the assessment of the UPDRS score requires notable effort and the available capacity of medical personnel is a bottleneck, under realistic conditions, the total UPDRS score is measured with a relatively low frequency, e.g., at the beginning of the treatment and after several months.

Little et al. (2009), Tsanas et al. (2010), and Sakar et al. (2013) have shown that the UPDRS score is related to various characteristics of the voice; thus, at least in theory, it could be estimated based on a patient’s speech while he/she makes telephone or Skype calls using his/her smartphone or tablet. With our current study, we would like to take a step toward this visionary application which, in the long term, is expected to allow continuous monitoring of the patient’s UPDRS score and almost immediate identification of its substantial changes.

One of the major challenges associated with the aforementioned visionary application is the fact that the exact function describing how the UPDRS score depends on (the combination of) quantifiable characteristics of speech is unknown. Therefore, state-of-the-art solutions for the estimation of UPDRS scores from voice data are based on machine learning (Sakar et al. 2013; Tsanas et al. 2011). Following the machine learning paradigm, voice data may be collected from a large set of patients’ audio recordings and UPDRS scores at the time of recording. Such data allows machine learning approaches to “discover” the dependency between the characteristics of the voices and the UPDRS scores so that the UPDRS scores of “new” patients can be estimated based on their speech.

Because artificial neural networks (ANNs) are known to be universal approximators (Pang-Ning, Steinbach, and Kumar 2006), we base our solution on ANNs. In particular, after studying the hubness phenomenon and the presence of bad hubs in the context of the UPDRS score estimation in a later section, we propose hubness-aware error correction for ANNs in order to increase the accuracy of estimation. To the best of our knowledge, the current work is the first attempt to exploit hubness in the context of ANNs. We perform experiments on publicly available datasets derived from real-voice data and show that the proposed error correction technique systematically increases the accuracy of various feedforward neural networks.

Background

In order to ensure that this article is self-contained, we provide the most relevant background information about the UPDRS in the following subsection and then review related works and basic definitions used throughout the article, respectively, in the two subsections that follow that.

Unified Parkinson’s disease rating scale

The Unified Parkinson’s Disease Rating Scale (UPDRS) was developed to provide a comprehensive coverage of the symptoms, in order to allow for clinical examination and follow-up of the progression of the disease. Today it serves as the gold-standard reference scale.

The scale has four parts. Part I (previously titled Mentation) was designed to assess nonmotor experiences of daily living. Part II (previously called Activities of Daily Living) assesses motor experiences of daily living. Part III (a.k.a. the Motor part) contains the examination of the patient’s motor skills, and Part IV (titled as Complications) considers motor complications.

The aforementioned parts of the UPDRS are measured at different frequencies; for example, according to Goetz, Poewe, and Rascol (2003), Part III was used in 98% of the cases, whereas Part I was used with a frequency of 60% only. For more details about the UPDRS score, the reader is referred to Goetz et al. (2008).

Alteration of speech is a well-known symptom of PD, about 70% of PD patients exhibit speech impairment (Hartelius and Svensson 1994; Logemann et al. 1978). Speech disturbances are represented in Parts II and III of the UPDRS. Speech disturbances in PD are characterized by hypophonia, hypokinetic dysarthria, palilalia, and speech dysfluency. With the progression of the disease, due to the involvement of speech organs, worsening of speech is known. Moreover, “positive effect of L-dopa treatment on speech disorders could be objectively confirmed” (Pawlukowska et al. 2015). In this work, we aim to estimate patients’ UPDRS scores from voice measures.

Related works

Machine learning techniques are widely applied for medical tasks (Cyganek and Wozniak 2015; Grana et al. 2011; Froelich, Wrobel, and Porwik 2015).

As we formalize the task of automated estimation of UPDRS score as a regression task, when reviewing related works, we focus on regression, which is one of the most prominent fields of machine learning with various applications in medicine (Busra Celikkaya et al. 2013; Soyiri, Reidpath, and Sarran 2013). In the last decades, various regression techniques have been developed, ranging from simple linear and polynomial regression over nearest neighbor regression, to more complex models, such as artificial neural networks (ANNs) and support vector regression (Devroye et al. 1994; Adamczak, Porollo, and Meller 2004; Basak, Pal, and Patranabis 2007).

One of the most interesting recent observations is the presence of hubs in various datasets. Informally, hubs are instances that are similar to a surprisingly high number of other instances. Unfortunately, some of the hubs are bad in the general sense that they could mislead machine learning algorithms. The presence of hubs has been studied and surveyed primarily in the context of classification, clustering, and instance selection (Radovanović, Nanopoulos, and Mirjana 2010a, 2010b; Tomašev and Dunja 2013; Radovanović, Nanopoulos, and Ivanović 2009; Tomašev et al. 2011; Tomašev, Radovanović, et al. 2015; Buza, Nanopoulos, and Schmidt-Thieme 2011; Tomašev, Buza, et al. 2015).

To the best of our knowledge, Buza, Nanopoulos, and Nagy (2015) were the first to study the presence of hubs in regression tasks. They focused on nearest neighbor regression and considered various applications, whereas in the subsequent sections of this article, we study the role of hubs in the estimation of the UPDRS score and propose a hubness-aware enhancement of ANNs.

Definitions and notations

A dataset Δ containing n instances is given. In our case, each instance corresponds to an audio recording. Numeric features describing characteristic properties of the voice are extracted; therefore, each instance is a vector of such features. Instances are denoted by . For each instance , the value of the continuous target, i.e., UPDRS score, is given and it is denoted by . We say that is the label of instance and Δ is the training dataset. With regression, we mean the task of predicting (estimating) the label of an instance .

We use to denote the distance between two instances and . In order to study the hubness phenomenon, we will use the notion of -nearest neighbors of an instance , which is a subset of so that and

We may omit the upper index Δ whenever there is no ambiguity. We note that ties may be broken arbitrarily, i.e., in the case that several subsets fulfill the above condition, any of them may be used as the set of nearest neighbors.

Bad hubs in UPDRS score estimation

Informally, hubness in datasets refers to the phenomenon that some instances are similar to a surprisingly large number of other instances. In order to quantitatively study hubness in context of UPDRS score estimation from voice data, we use the notion of k-nearest neighbors.

Let us first note that the k-nearest neighbor relationship is asymmetric: although each instance has k nearest neighbors, an instance does not necessarily appear k times as one of the k-nearest neighbors of other instances. This is illustrated in Figure 1a for k = 1. In order to keep the example simple, we consider two-dimensional vector data, therefore, instances correspond to points of the plane. In the context of UPDRS score estimation from speech data, we may imagine a simple scenario in which two numeric features of the audio signals (such as shimmer and jitter) are extracted and we use only these two features to represent the data. Each of these features may correspond to either the horizontal or vertical axis, thus, the audio recordings may be mapped to points in the plane.

Figure 1. (a) The nearest neighbor relationship is asymmetric. Some instances never appear as the first nearest neighbor of other instances, although there are some instances that appear frequently as the first nearest neighbor of other instances. (b) Example used to illustrate error correction.

In Figure 1a, there is a directed edge from each instance (denoted by a circle) to its first nearest neighbor. Whereas each instance has exactly one first nearest neighbor, how many times an instance appears as the first nearest neighbor of other instances (i.e., the number of incoming edges to an instance) is not necessarily one. As can be seen, some of the instances never appear as nearest neighbors of others, and there is an instance that appears as the first nearest neighbor of three other instances. In particular, the integer next to each instance shows how many times it appears as the first nearest neighbor of others.

Generally, we use to denote how many times the instance appears as one of the k-nearest neighbors of other instances of Δ. It is easy to see that the expected value of is , however, the actual value of varies from instance to instance. As was shown by (Radovanović et al. 2010a; Buza et al. 2011; Tomašev and Dunja 2013) in many cases, the distribution of is substantially skewed to the right, i.e., there are a few instances with extraordinarily high values, furthermore, the skewness increases with increasing intrinsic dimensionality of the data. Usually, instances having surprisingly high are called hubs, and instances with exceptionally low are called antihubs. More precisely, we say that an instance is a hub, if ; and an instance is an antihub if . The phenomenon that is skewed is called hubness and it is often quantified by the third standardized moment (skewness) of the distribution of .

In order to show that there are instances that might mislead machine learning models, we perform the following analysis on the telemonitoring and multiple sound recording datasets, both of them containing voice data for UPDRS estimation. (The datasets are described in a later section in more detail.) We considered both estimation tasks (total and motor UPDRS score) associated with the telemonitoring data separately. For each instance , as , we calculate the average absolute difference between the label of (i.e., UPDRS score associated with ) and the labels of those instances that have as one of their nearest neighbors. Formally, let

(1)

Figure 2. The distribution of in the case of motor UPDRS scores of the telemonitoring dataset for (a) low-error instances, (b) high-error instances, and (c) both histograms in the same plot. Similar observations can be made for the multiple sound recording dataset and total UPDRS scores of the telemonitoring dataset as well. Note that some of the high-error instances appear as nearest neighbors of many other instances, i.e., there are bad hubs in the data. Remarkably, the distribution of high-error instances is shifted to the right compared with the distribution of low-error instances. This indicates that there are more high-error hubs than low-error hubs.

then

(2)

After calculating the above error for each instance, we ordered the instances according to their errors and selected 25% of the instances having highest error, and another 25% having lowest error. We call these instances high-error instances and low-error instances. Throughout the analysis, we used k = 10 and the Euclidean distance over all biomedical voice features present in the datasets. Figure 2 shows the distributions of for low- and high-error instances of the telemonitoring dataset in the case of motor UPDRS scores. In the figure, the horizontal axis corresponds to N₁₀ (x), while the height of the column shows how many instances have that particular value of N₁₀ (x).

As one can see, the distributions of N₁₀ (x) are notably skewed. Most importantly, some of the high-error instances appear as nearest neighbors of many other instances. In particular, as we defined hubs as instances that appear as nearest neighbors of more than 2k instances, we observe that there are hubs among the high-error instances. We use the term bad hubs to refer to hubs among high-error instances. Additionally, let us note that the distribution of high-error instances is shifted to the right compared with the distribution of low-error instances. This indicates that there are more bad hubs than low-error (or good) hubs.

Hubs tend to be located in dense regions of the data space; according to recent results, they might even serve as cluster centers (Tomašev et al. 2015b). Under the assumption that the model will be applied to instances originating from the same (or at least similar) distribution as the distribution from which the training data originates, it is essential for any regressor to perform well on instances being “close to” hubs, because many of the new/test instances are expected to be located exactly in these regions, i.e., in the proximity of hubs. Therefore, in the next section, we devise a mechanism that is able to compensate for the detrimental effect of high-error instances, including high-error instances located at “central” positions, i.e., bad hubs.

Hubness-aware artificial neural networks

Next, we describe error correction, a mechanism that can be used to improve the performance of ANNs. We define the corrected label of an instance x as

(3)

where denotes the set of instances that have as one of their -nearest neighbors; see Equation (1) for the formal definition of . We propose to use the corrected labels instead of the original labels while training ANNs. Although our current work focuses on ANNs, we note that, in principle, the above error correction technique may be used with various other regressors as well.

Using the example in Figure 1b, we illustrate how the corrected labels are calculated. In Figure 1b, training instances are denoted by circles. They are identified by the symbols . The numeric value next to each instance shows its label. In order to keep the example simple, we use to calculate the corrected labels of training instances. For training ANNs, the corrected label of all the training instances need to be calculated, however, we present the calculations only for and because the procedure is the same in the case of the other instances as well. Concretely, the corrected labels of and are

Experiments

In this section, we present the results of our experimental evaluation of the proposed approach on two real-world speech datasets associated with UPDRS scores as prediction targets.

Datasets

The Parkinson’s Telemonitoring dataset (Little et al. 2009; Tsanas et al. 2010) “is composed of a range of biomedical voice measurements from 42 people with early-stage Parkinson’s disease recruited to a six-month trial of a telemonitoring device for remote symptom-progression monitoring.”² In total, the data contains 5875 instances. Both motor (i.e., part III) and total (i.e., all four parts) UPDRS scores as well as temporal information (i.e., on which day the measurements were taken) are available. We use motor to denote the experiments when the motor UPDRS score was used as target; analogously, total denotes the experiments when the total UPDRS score was used as target.

We performed experiments on the Parkinson Speech Dataset with Multiple Types of Sound Recordings as well, to which we refer as multi for simplicity (Sakar et al. 2013). This dataset contains 1040 instances.

Both datasets are available in the UCI Machine Learning repository (Bache and Lichman 2013). For both datasets, we used jitter and shimmer features.

Experimental protocol

We performed experiments according to the patient-based 10 × 10-fold cross-validation protocol, i.e., in each round of the 10 × 10-fold cross-validation, all the instances belonging to the same patient either appear in the train or test split. This simulates the medically relevant scenario in which historical data is used to train the model, which is then applied to the estimation of UPDRS scores of new patients.

Because temporal information was available in the telemonitoring dataset, an additional estimation task that may be associated with the data is to estimate the change of UPDRS score relative to its initial value. Because it might be highly relevant to identify substantial changes in the UPDRS score of a patient as early as possible, we also evaluated the proposed technique in the UPDRS score change estimation context. These experiments are denoted as change motor and change total, respectively.

In all the aforementioned experiments, features were normalized by subtracting their mean and dividing by their standard deviation. Mean and standard deviation were calculated on the training subset of the data (therefore, the normalization was performed in each round of the 10 × 10-fold cross-validation).

In our experiments, we used the implementation of ANNs from the Weka software package (Witten and Frank 2005). In particular, we used feedforward ANNs trained with backpropagation with learning rate of 0.3 and momentum of 0.2, and the number of training epochs was set to 500. In order to show that the proposed approach is indeed able to improve the performance of neural networks, we used neural networks of the same structure both with and without the error correction technique described in previously. While performing error correction, we used the Euclidean distance and as the default -value in order to calculate the nearest neighbor relationship. Error correction was performed only on the training set, i.e., in both cases (with and without error correction) we aimed to predict the same target.

We use a comma-separated list of integers to denote the structure of ANNs: each item of the list corresponds to one of the hidden layers and it denotes the number of neurons in that layer. For example, “Net-5,5” denotes an ANN with two hidden layers, each of them containing 5 neurons. We performed experiments with ANNs of six different structures: one hidden layer with (i) 5, (ii) 10, and (iii) 20 neurons and two hidden layers each of them having (iv) 5, (v) 10, and (vi) 20 neurons respectively.

Performance metrics

We measured the performance of our approach and the baselines in terms of mean absolute error (MAE) and root mean square error (RMSE):

where and denote the test set and its size, respectively, denotes the label predicted for instance , while denotes the true label of instance . We used paired calibrated t-tests proposed by Bouckaert (2003) at significance level of 0.05 to examine if the ANNs with error correction statistically significantly outperform ANNs without error correction. For simplicity, we report results only for MAE, but we note that we observed similar trends for RMSE as well.

Results

In each of the five different contexts (i.e., motor, total, multi, change motor and change total), we examined six types of neural networks, this gave, in total, 30 experiments. In each of these 30 experiments, we compared neural networks using error correction (+EC) and neural networks without error correction (EC).

Table 1 summarizes the results. We report MAE averaged over 10 10 folds. As one can see, the proposed error-correction mechanism systematically improves the quality of UPDRS score estimation. In particular, in all of the aforementioned 30 experiments, the neural networks with error correction outperformed the neural networks without error correction. The improvement is statistically significant in 24 experiments.

Table 1. Mean absolute error averaged over 10 × 10 folds with (+EC) and without error correction (–EC). The MAE-value is underlined if the difference is statistically significant.

	Motor	Total	Multi	Change motor	Change total
Model	+EC	−EC	+EC	−EC	+EC	−EC	+EC	—EC	+EC	–EC
Net-5	7.206	7.669	9.477	10.080	13.944	14.139	3.150	3.445	3.830	4.117
Net-10	7.232	7.687	9.504	10.056	13.968	14.548	3.148	3.469	3.827	4.127
Net-20	7.276	7.822	9.594	10.065	14.104	14.505	3.136	3.440	3.824	4.109
Net-5,5	7.220	7.558	9.451	10.035	13.764	14.114	3.103	3.473	3.716	4.022
Net-10,10	7.239	7.572	9.476	10.025	13.797	14.145	3.104	3.451	3.718	4.004
Net-20,20	7.266	7.589	9.452	10.014	13.852	14.260	3.106	3.434	3.712	4.013

As noted previously, error correction may be used with various models. Therefore, we examined the effect of error correction on other regression techniques as well. Figure 3a shows the performance of various models in the case of estimating the motor UPDRS score with and without error correction. In particular, we consider: (1) Net-5, feedforward artificial neural network with one hidden layer containing five neurons, (2) k-NN, k-nearest neighbor regression with k = 10, (3) M5P regression trees from the aforementioned Weka machine learning library, and (4) SVM, support vector regression with linear kernel. As one can see, error correction is able to improve the performance of these models as well.

Figure 3. (a) Performance (MAE) of various models when predicting the motor UPDRS score with (EC) and without (EC) error correction. (b) Performance of Net-5 with error correction (EC) using various -values, and without error correction (EC).

As described, in order to perform error correction, -nearest neighbor relationships are computed first. Figure 3b shows the performance of Net-5 when performing error correction with various k-values. As can be seen, error correction systematically improves the performance of the model for all the examined k values, and k values between 5 and 10 are preferable.

Discussion

During spontaneous usage of tablets, smartphones, and laptops, these devices are able to capture unprecedented amounts and varieties of data about their users. The gathered information might range from the dynamics of typing recorded while the user writes short messages or e-mails, over voice recordings performed during phone or Skype calls, to GPS coordinates and social connectivity features (such as how many people the user regularly writes messages to). It is hypothesized that such information might be related to various disorders or predict unexpected changes of the user’s health conditions (Estrin 2013). Given the relatively high and increasing computational power of smartphones and tables, processing and analysis of the data collected during spontaneous usage became technologically possible. This is expected to give rise to visionary health-care applications that support medical doctors in diagnostic decisions and treatment of diseases.

Although much of the research focuses on understanding the background of PD, (Zimprich et al. 2003; Balicza et al. 2012), in this article, we focused on the estimation of UPDRS score from biomedical voice measurements. Due to the fact that the user’s voice can be simply recorded during spontaneous interactions with tablets or smartphones, we strongly hope that our work is a step toward the usage of tablets and smartphones in the strict follow up of PD. Such visionary applications, once they will be realized, will not only allow for inexpensive and continuous monitoring of patient status, but the resulting data is expected to increase the patients’ and their relatives’ awareness of the disease, which is one of the key factors of successful treatment of the disease.

We hope that, over the long term, similar approaches will be used for diagnosis and monitoring of other diseases, as well. For example, Estrin (2013) reported that reduction of hearing abilities could be detected by examining the volume at which the user listened to music on his smartphone. This increased the patient’s awareness of the disease and convinced the patient to turn to medical doctors for examination. Dementia and cognitive impairment (Bereczki and Szatmári 2009), is another domain in which telemonitoring systems might be advantageous over the long term.

With regard to the limitations of our study, we have to mention that it is likely that the accuracy of UPDRS score estimation needs to be increased further for successful applications. Therefore, incorporation of further data mining techniques, such as monotonization (Horváth et al. 2011) and monotone models (Horváth and Vojtáš 2006) and the adaptation of hybrid solutions (Woźniak, Graña, and Corchado 2014) might be advantageous. We also mention that the proposed error-correction technique could contribute to avoiding overfitting to hub instances having nonrepresentative labels. In contrast, conventional regularization techniques focus on model complexity and treat all the instances as equally important. More detailed study of the relation between regularization and error correction is left for future work.

The data recorded during spontaneous usage of smartphones and tablets was originally not designed for diagnostic purposes, therefore, even the most useful pieces of the data can be expected to be only weakly correlated with medically relevant conditions. Note, however, that a combination of weak features can serve as a reasonable predictor even if the features are weak predictors separately, and, by training neural networks, the machine is expected to learn the appropriate combination of those individually weak predictors. However, medical doctors are expected to play a crucial role in the correct interpretation of the data and potential additional examinations. The overall workload of medical doctors will not necessarily decrease: although continuous telemonitoring might replace some of the scheduled face-to-face meetings between patients and doctors, due to the continuous monitoring of UPDRS scores or other conditions and possible false alarms generated by automated recognition systems, patients might ask for more appointments with medical doctors.

Conclusions

In this article, we focused on the automated estimation of UPDRS scores based on biomedical voice measures. This is a crucial component of telemonitoring systems for PD patients. We studied the hubness phenomenon in context of the UPDRS score estimation and proposed a hubness-aware error correction for artificial neural networks. We performed experiments on publicly available real-world datasets and showed that the proposed technique systematically improves estimation accuracy as measured by MAE and RMSE.

Funding

This research was performed within the framework of the grant of the Hungarian Scientific Research Fund – OTKA PD 111710. This article was supported by the János Bolyai Research Scholarship of the Hungarian Academy of Sciences. N. Á. Varga was supported by the KTIA NAP 13 1-2013-0001.

Additional information

Funding

This research was performed within the framework of the grant of the Hungarian Scientific Research Fund – OTKA PD 111710. This article was supported by the János Bolyai Research Scholarship of the Hungarian Academy of Sciences. N. Á. Varga was supported by the KTIA NAP 13 1-2013-0001.

Notes

1. http://www.pdf.org/en/parkinson_statistics

2. http://archive.ics.uci.edu/ml/datasets/Parkinsons+Telemonitoring