Clinical measures of communication limitations in dysarthria assessed through crowdsourcing: specificity, sensitivity, and retest-reliability

ABSTRACT

Assessing the impact of dysarthria on a patient’s ability to communicate should be an integral part of patient management. However, due to the high demands on reliable quantification of communication limitations, hardly any formal clinical tests with approved psychometric properties have been developed so far. This study investigates a web-based assessment of communication impairment in dysarthria, named KommPaS. The test comprises measures of intelligibility, naturalness, perceived listener effort and communication efficiency, as well as a total score that integrates these parameters. The approach is characterized by a quasi-random access to a large inventory of test materials and to a large group of naïve listeners, recruited via crowdsourcing. As part of a larger research program to establish the clinical applicability of this new approach, the present paper focuses on two psychometric issues, namely specificity and sensitivity (study 1) and retest-reliability (study 2). Study 1: KommPaS was administered to 54 healthy adults and 100 adult persons with dysarthria (PWD). Non-parametric criterion-based norms (specificity: 0.95) were used to derive a standard metric for each of the four component variables, and corresponding sensitivity values for the presence of dysarthria were identified. Overall classification accuracy of the total score was determined using a ROC analysis. The resulting cutscores showed a high accuracy in the separation of PWD from healthy speakers for the naturalness and the total score. Study 2: A sub-group of 20 PWD enrolled in study 1 were administered a second KommPaS examination. ICC analyses revealed good to excellent retest reliabilities for all parameters.

KEYWORDS:

• Dysarthria
• communication limitation
• web-based assessment
• crowdsourcing
• psychometric evaluation

Acknowledgments

This work was funded by the German Academic Scholarship Foundation and the Bayerische Sparkassenstiftung. We are grateful to Jakob Pfab and Klaus Jänsch for web development and technical support. We wish to thank all patients, volunteers, and anonymous crowdworkers for participating in this research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Ethics

Ethical approval (Project No. 19-365) was obtained from the Ethics Committee of the Faculty of Medicine at the Ludwig-Maximilians-University Munich, Germany. Participants were informed in detail about the study by the examiner and gave their written consent to participate.

KommPaS Study Group (clinical collaborators)

Madleen Klonowski^b, Nadine Geißler^b, Franziska Ammer^c, Christina Kurfeß^c, Holger Grötzbach^c, Alexander Mandl^d, Felicitas Knorr^d, Katrin Strecker^d, Theresa Schölderle^d, Sina Matern^e, Christiane Weck^e, Berthold Gröne^f, Stefanie Brühl^g, Christiane Kirchner^g, Ingo Kleiter^h, Ursula Sühn^h, Joachim von Eichmannⁱ, Christina Möhrle^j, Pete Guy Spencer^j, Rüdiger Ilg^k, Doris Klintwort^k, Daniel Lubecki^l, Steffy Marinho^m, Katharina Hogrefe^m)

^bSchön Klinik München Schwabing

^cAsklepios Klinik Schaufling

^dStiftung ICP München

^eKrankenhaus Agatharied

^fKliniken Schmieder Allensbach

^gSt. Mauritius Therapieklinik Meerbusch

^hMarianne-Strauß-Klinik Berg

ⁱm&i-Fachklinik Enzensberg

^jHegau-Jugendwerk Gailingen

^kAsklepios Stadtklinik Bad Tölz

^lNeuroKom Bad Tölz

^mMutabor e.V.

Notes

¹ More recently, Vojtech et al. (2019) examined communication efficiency in an experimental study of synthesized speech that might inform future developments of augmentative and alternative communication devices.

² The BoDyS evaluations for the patient sample were completed by three staff members (among them the first author) with specific training in the administration of the BoDyS. The reliability of the BoDyS scales was approved in Ziegler et al. (2017).

³ ‘Der Englische Garten ist einer der größten Parks der Welt. Er liegt in München, der Hauptstadt von Bayern. Im Park gibt es lange Spazierwege und mehrere Biergärten.‘ [The English Garden is one of the largest parks in the world. It is located in Munich, the capital of Bavaria. In the park there are long walks and several beer gardens]

⁴ They represent a subgroup of over 250,000 German speaking crowdworkers registered at Clickworker. Clickworker states that their community is a broad cross-section of the population in terms of age or employment level. Furthermore, the distribution of gender with 51% men and 49% women is roughly balanced (Clickworker, 2021).

⁵ Speech rate is measured automatically during the upload of the recording by detecting the start and end positions of the spoken sentences using the software ffmpeg and calculating the individual durations of the speech samples. The start and end positions are manually reviewed and corrected in case of errors.

⁶ In this article, only raw scores are reported for the four component variables of the KommPaS profile. Standardized scores are only used for the computation of the total score.

⁷ The ROC analysis was performed using the R-package ‘pROC’ (Robin et al., 2011).

Additional information

Funding

This research was supported by a PhD fellowship from the Studienstiftung des Deutschen Volkes awarded to the first author and a grant from the Bayerische Sparkassenstiftung.;