Automated recognition of functioning, activity and participation in COVID-19 from electronic patient records by natural language processing: a proof- of- concept

Figures & data

Table 1. COVID-19 relevant ICF categories and levels.

	Category
Level	Walking and Moving	Exercise tolerance	Emotional functions	Work and employment
0	No ability. The patient cannot walk or needs help from two or more people to walk or walks in a walkway.	Equivalent to 0 ≤ MET < 1. Only lying activities can be sustained physically	(Very) anxious, gloomy, angry, sad, unstable, tense, etc. and/or very often/quickly occurring mood switches.	The patient is not able to work/study.
1	Dependent (first degree). The patient needs continuous solid support from a person to bear weight and maintain balance.	Equivalent to 1 ≤ MET < 2. Activities while lying and sitting can be sustained.	A little anxious, sad, angry, sad, etc. and/or often occurring mood switches.	Very limited ability to work/study
2	Dependent (second degree). The patient needs continuous or intermittent assistance in maintaining balance or coordination.	Equivalent to 2 ≤ MET < 3. Walking at a slow to moderate pace can be physically sustained, as well as shopping and small household tasks.	Neutral.	The patient is able to work/study for about 50% of their full capacity when healthy, or the patient is only able to work/study at home and is not able to go to school/the office.
3	Supervision. The patient needs supervision from a person for safety and needs at most verbal guidance while walking. However, the patient does not need physical contact to walk.	Equivalent to 3 ≤ MET < 4. Walking and/or cycling at a normal pace, gardening and exercises without equipment are possible.	A little cheerful, positive, happy, content, stable, etc.	The patient is able to work/study at almost their full capacity when healthy.
4	Independent (limited). The patient can walk independently on a flat surface, but cannot safely climb stairs, climb slopes, or walk on uneven surfaces.	Equivalent to 4 ≤ MET ≤ 6. Cycling and/or walking at a high pace, considerable exercises such as cycling from 16 km/h and heavy housework can be physically sustained.	(Very) cheerful, positive, happy, content, stable, etc.	The patient is able to work/study at full capacity when healthy.
5	Independent. The patient can walk independently on a flat surface, on uneven surfaces, on slopes and can climb stairs.	Equivalent to MET > 6. Jogging, strenuous exercise, running, climbing stairs quickly, and sports can be sustained.	–	–

Figure 1. Overview of the annotation data divided into non-COVID-19 and COVID-19 notes for each annotated ICF category. Each category is differentiated in levels (0–4 and in the case of Walking & Moving and Exercise Tolerance 0–5).

Figure 2. Distribution of level annotations for Walking & Moving (FAC score) across COVID-19 and non-COVID-19 data.

Table 2. Performance of the ICF category classification.

	Non-COVID-19 test set			COVID-19 test set
	P	R	F1	P	R	F1
Walking & moving	0.723	0.661	0.691	0.733	0.629	0.677
Emotional functions	0.771	0.646	0.703	0.545	0.476	0.508
Exercise tolerance	0.676	0.247	0.356	0.667	0.240	0.353

Work and Employment were omitted because of the lack of data in the COVID-19 annotations. P: precision; R: recall; F1: harmonic mean. Models NOT trained on combined datasets (training data of COVID-19 and non-COVID-19) are in bold. If a score is in bold, it means that it was trained only on the type of data that it was tested on (trained on non-COVID-19 and tested on non-COVID-19, for example). All the best models are evaluated at the note level.

Table 3. Best models regression analysis for each relevant scoring level.

	Non-COVID-19			COVID-19
	MSE	MAE	RMSE	MSE	MAE	RMSE
Walking & moving	1.65	0.81	1.28	1.37	0.91	1.17
Emotional functions	0.44	0.47	0.66	0.67	0.64	0.82

Exercise Tolerance and Work and Employment were omitted because of the lack of data in the COVID-19 annotations.

MSE: Mean Squared Error; MAE: Mean Absolute Error; RMSE: Root Mean Squared Error.

Figure 3. Impact of the size of the training data on performance on COVID-19 data. Per category, grouped bars from top to bottom show the magnitude of precision (P), recall (R) and their harmonic mean (F1) as a function of increasing sample size.

Data availability statement

The data that support the findings of this study are available on request from the corresponding author, CGM. The data are not publicly available due to containing information that could compromise the privacy of research participants.