The Performance of Digital Monitoring Devices for Oxygen Saturation and Respiratory Rate in COPD: A Systematic Review

Abstract

Healthcare access and delivery for individuals with chronic obstructive pulmonary disease (COPD) who live in remote areas or who are susceptible to contracting communicable diseases, such as COVID-19, may be a challenge. Telehealth and remote monitoring devices can be used to overcome this issue. However, the accuracy of these devices must be ensured before forming healthcare decisions based on their outcomes. Therefore, a systematic review was performed to synthesize the evidence on the reliability, validity and responsiveness of digital devices used for tracking oxygen saturation (SpO₂) and/or respiratory rate (RR) in individuals with COPD, in remote settings. Three electronic databases were searched: MEDLINE (1996 to October 8, 2020), EMBASE (1996 to October 8, 2020) and CINAHL (1998 to October 8, 2020). Studies were included if they aimed to evaluate one or more measurement properties of a digital device measuring SpO₂ or RR in individuals with COPD. Six-hundred and twenty-five articles were identified and after screening, 7 studies matched the inclusion criteria; covering 11 devices measuring SpO₂ and/or RR. Studies reported on the reliability (n = 1), convergent validity (n = 1), concurrent validity (n = 2) and predictive validity (n = 2) of SpO₂ devices and on the convergent validity (n = 1), concurrent validity (n = 1) and predictive validity (n = 1) of RR devices. SpO₂ and RR devices were valid when compared against other respiration monitoring devices but were not precise in predicting exacerbation events. More well-designed measurement studies are needed to make firm conclusions about the accuracy of such devices.

Supplemental data for this article is available online at https://doi.org/10.1080/15412555.2021.1945021 .

Keywords:

• Digital devices
• oxygen
• respiratory rate
• COPD
• reliability
• validity

Introduction

Chronic obstructive pulmonary disease (COPD) is characterized by airflow limitation resulting from lung abnormalities (e.g. alveolar destruction, bronchial inflammation and excess mucous production), typically brought about by significant exposure to tobacco and/or pollutants [1]. These lung abnormalities cause individuals to experience difficulties breathing and taking in oxygen, with dyspnea being the most common symptom [1]. Many treatments, such as pulmonary rehabilitation or oxygen therapy, can improve breathing and oxygen intake and reduce symptoms [2,3]. However, the accessibility of these treatments may be difficult for individuals living in remote areas (i.e. areas distant from hospitals and/or clinics) or for those who are unable to visit a healthcare provider or facility because of high-risk susceptibility to communicable diseases. The current physical distancing restrictions implemented in Canada and across the world, resulting from the COVID-19 pandemic, may also prohibit access to in-person care. Furthermore, individuals with chronic health conditions, such as cardiovascular conditions or COPD, are at higher risk of infections and complications from COVID-19 [4,5], which emphasizes their need to follow public health guidelines. Moreover, individuals with COPD tend to experience periods of acute worsening of symptoms (i.e. exacerbations) resulting in healthcare utilization or in some cases, hospitalization [6]. Exacerbations can usually be prevented through managing and monitoring patients’ symptoms [6], however, due to physical restrictions (e.g. living in remote areas) or fear of exposure to viruses (e.g. COVID-19), individuals may avoid seeking healthcare services [7].

Fortunately, with current advances in technology, the delivery of healthcare services to individuals is viable. For example, the delivery of pulmonary telerehabilitation for individuals with COPD has been a feasible and effective option for improving exercise tolerance and quality of life [8–10]. Since respiration is the most impacted activity in COPD, remote monitoring of oxygen intake and breathing can be valuable for healthcare professionals when assessing virtual (e.g. pulmonary telerehabilitation) and non-virtual treatment outcomes and monitoring patients’ health status. A common digital device for monitoring oxygen saturation (SpO₂) is a pulse oximeter [11], and common digital devices used for tracking respiratory rate (RR) are plethysmographs [12]. These devices can be equipped with Bluetooth capabilities and be easily adopted in remote care settings. To ensure these respiration telemonitoring devices are accurate and consistent in reporting respiration outcomes (i.e. SpO₂ or RR) and before recommendations can be made as to which device should be widely adopted in clinical practice, their measurement properties (i.e. reliability, validity and responsiveness) must be evaluated [13]. Therefore, a systematic review was performed to synthesize the evidence on the reliability, validity and responsiveness of digital devices used for tracking SpO₂ and/or RR in individuals with COPD, in remote settings.

Methods

This review has been registered with PROSPERO (registration number: CRD42020219734).

Search strategy

This review followed Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) guidelines [14] for evaluating and reporting measurement properties. The following electronic databases were searched: MEDLINE (1996 to October 8, 2020), EMBASE (1996 to October 8, 2020), and CINAHL (1998 to October 8, 2020). Search terms were a combination of (1) the population (COPD), (2) digital technology terms (e.g. mobile applications, technology, digital, virtual, smartphone, software), (3) outcome (i.e. oxygen, breathing, respiration) and (4) measurement properties (using the search filter developed by Terwee et al. [15]). A comprehensive list of all the search terms is provided in Appendix 1. Both medical subject headings and keyword terms were used in the search (medical subject headings used if available). Two independent reviewers (AM and EW) screened titles, abstracts and full text.

The inclusion criteria for studies were: 1) the study assessed a digital device (a device that can store and transmit digital information) that aimed to measure SpO₂ or RR remotely (i.e. designed for home or virtual use); 2) the study sample represented the population of interest (at least 70% had a clinical diagnosis of COPD); and 3) the aim of the study was the evaluation of one or more measurement properties of the device. Articles were excluded if they were: 1) not peer-reviewed; 2) not in English; 3) gray literature (e.g. meeting/conference proceedings/abstracts); and 4) reviews.

Data extraction

Data extraction was performed by two independent reviewers (AM and EW) to avoid missing any information. Study characteristics, such as country, setting, sample size, age, sex, forced expiratory volume (FEV₁) and pack-years, were extracted if available. The following measurement properties were extracted: reliability, construct validity (convergent and known-groups validity), criterion validity (concurrent and predictive validity) and responsiveness (Table 1).

Table 1. Measurement properties (definitions based on De Vet et al. [13]).

Measurement property	Definition
Reliability	the extent to which two or more scores are consistent for individuals, under stable conditions, at different time points
Construct validity
Convergent validity	the extent to which two instruments (measuring a related construct) associate
Known-groups validity	the extent to which an instrument can differentiate between individuals with differing characteristics
Criterion validity
Concurrent validity	the extent to which an instrument associates with the ‘gold standard’ (i.e. well-accepted measures in the field)
Predictive validity	the extent to which an instrument can measure a future outcome
Responsiveness	the extent to which an instrument can detect change in a construct over time

Quality assessment

The COSMIN risk of bias checklist was utilized by the research team to assess the methodological quality of each study [16]. The checklist includes separate criteria for each measurement property. The following boxes were examined during evaluation: Box 6. Reliability, Box 8. Criterion validity and Box 9. Hypothesis testing for construct validity. The methodological quality of each measurement property was evaluated through rating items, concerning the study design and statistical methods utilized, as very good, adequate, doubtful or inadequate. The lowest rating for the items was used to determine the overall rating of the property. This process was performed by the two independent reviewers (AM and EW) and discrepancies were discussed until consensus was reached.

COSMIN’s criteria for good measurement properties was used to rate the result of each measurement property per device [14]. A rating of sufficient, insufficient or indeterminate was given according to previously defined hypotheses formulated by the research team using COSMIN guidelines (Appendix 2). The research team defined hypotheses a priori and rated results according to them (regardless of the hypotheses defined by the authors) to ensure all results were comparable to the same set of hypotheses [14]. A sufficient rating was given if the hypothesis was met and an insufficient rating was given if it was not met. If the hypothesis was not defined a priori or the measurement value was not reported, an indeterminate rating was given. This process was performed by the two independent reviewers and discrepancies were discussed until consensus was reached. For reliability, intraclass correlation coefficients (ICCs) were expected to be greater or equal to 0.70 to meet the sufficient rating cutoff [14]. For validity, areas under the curve (AUCs) were hypothesized to be greater or equal to 0.70 for acceptable accuracy [14]. For studies using Bland-Altman plots, results were rated based on hypotheses defined a priori by the authors since this form of analysis is dependent on the sample [17]. Statistical significance for all studies was set at p < 0.05. Kappa statistics were interpreted as follows: ≤ 0 = no agreement, 0.01–0.20 = slight agreement, 0.21–0.40 = fair agreement, 0.41– 0.60 = moderate agreement, 0.61–0.80 = substantial agreement and 0.81–1.00 = almost perfect agreement [18].

Quantitative analysis

For results (per measurement property, per outcome) that could be statistically pooled and were at least of adequate quality [13,14], forest plots were created to demonstrate the overall validity of the devices. Mean differences and 95% limits of agreement (LoA) were pooled using the fixed-effects approach. The statistical heterogeneity (i.e. % variation not due to chance) between the devices was measured using the I² statistic, with higher values representing greater heterogeneity [19]. Stata, version 15.1 (StataCorp, College Station, TX, USA) was used to perform analyses.

Results

Selection process

A total of 625 articles were identified through searching the databases. After removing duplicates (n = 101) and irrelevant studies (n = 473), 51 full-text articles were screened. Forty-four articles were excluded because 1) the outcome of the study or the measure utilized was not a digital device aimed to measure SpO₂ or RR remotely, (2) they were conference proceedings or abstracts, (3) they were not a measurement study or did not examine any measurement components of the device, (4) the target population was not individuals with COPD or (5) they were not in English. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flow diagram was used to display the number of records included in this review (Figure 1).

Figure 1. PRISMA [20] flow diagram of included studies.

Summary of studies

Seven studies were included; representing 11 devices. Four devices measured SpO₂ [21–24], 6 measured RR [25,26] and 1 measured both SpO₂ and RR [27]. A total of 201 individuals with COPD were included in this review, with sample sizes ranging from 5 to 110 participants, with the majority of studies (i.e. 5 out of 7) including 10-20 participants. The mean age of participants in the included studies ranged from 67 to 73 and the mean FEV₁% predicted ranged from 44 to 63.5. Studies were performed either remotely, in participants’ homes (n = 3), during pulmonary rehabilitation sessions (n = 2) or in laboratory settings (n = 1). Supplemental Table S1 outlines the devices, the sociodemographic information of participants and the psychometric properties assessed in each included study.

Oxygen saturation (SpO₂) devices: measurement properties

Supplemental Table S2 outlines the methodological quality and measurement results for each study examining devices reporting on SpO₂.

Reliability

One study examined the reliability of SpO₂ devices. Chan et al. [22] examined the test-retest reliability of the Kenek O₂ pulse oximeter, the only smartphone oximeter approved by the Government of Canada for medical use, under two pulmonary rehabilitation activities: during (1) a cycle ergometer test and (2) a treadmill test. The ICC values obtained for both conditions did not meet the expected cutoff of 0.70 (ICC cycling = 0.54; ICC treadmilling= 0.33).

Convergent validity

One study evaluated the convergent validity of SpO₂ devices against other measures. Chan et al. [22] evaluated the agreement (using Bland-Altman plots) between the Kenek O₂ pulse oximeter and the Nonin 8500 pulse oximeter under three pulmonary rehabilitation conditions: (1) at rest, (2) during a cycle ergometer test and (3) during a treadmill test. At rest, the Kenek oximeter consistently reported 1.4% (standard deviation = 1.9) higher SpO₂ than the Nonin oximeter and had a narrow LoA (-2.4, 5.1). This mean difference fell within the acceptable range predefined by the authors (i.e. + or − 2%). On the other hand, during activity, the mean difference between the two devices fell outside the 2% range and their LoAs were wider.

Known-groups validity

No studies evaluated the known-groups validity of devices used for measuring SpO₂.

Concurrent validity

Two studies evaluated the concurrent validity of SpO₂ devices against gold standard methods (i.e. well-accepted methods in the field). Bonnevie et al. [23] explored the agreement between transmitted and locally stored SpO₂ data from a Nonin 3150 pulse oximeter during pulmonary rehabilitation, through a Bland-Altman plot and percent agreements. Transmitted data was consistently 0.13% higher than locally stored data (95% LoA=-0.297, 0.337). The mean exact agreement was 87% and the mean similar agreement (values within +/- 3%) was 100% between the two data sets. Cooper et al. [21] used Kappa statistics to evaluate agreement between SpO₂ less than 90% (measured by the Onyx II oximeter) and Anthonisen criteria for exacerbation. There was fair agreement between SpO₂ less than 90% and Anthonisen criteria 1 (i.e. an increase in dyspnea, sputum volume and sputum purulence, over two consecutive days), and moderate agreement between SpO₂ less than 90% and Anthonisen criteria 2 (i.e. an increase in at least two out of three symptoms: dyspnea, sputum volume, sputum purulence, over two consecutive days) and modified Anthonisen criteria (i.e. an increase in two major symptoms (dyspnea, sputum volume, sputum purulence) or an increase in one major symptom and one minor symptom (cough or wheezing), over two consecutive days) [28].

Predictive validity

Two studies evaluated the predictive validity of SpO₂ devices in detecting an exacerbation event. Shah et al. [27] defined an exacerbation as “an event in the natural course of the disease characterized by a change in the patient’s baseline dyspnea, cough, or sputum that is beyond normal day-to-day variations, is acute in onset, and may warrant a change in regular medication in a patient with underlying COPD”. The SpO₂ measured by a Bluetooth enabled pulse oximeter almost met the AUC acceptable cutoff of 0.70 to accurately predict an exacerbation event (AUC = 0.66). Burton et al. [24] defined an exacerbation as meeting the Anthonisen criteria [28] and taking antibiotics in the two-day period, which was transformed into a total symptom score. A pulse oximeter measuring SpO₂ was able to predict exacerbations compared to usual days with 13% higher odds, but not compared to ‘bad days’ (i.e. the criterion for exacerbation was met on one day and then fell below the threshold the following day) with 3% higher odds.

Responsiveness

No studies evaluated the responsiveness of devices used for measuring SpO₂.

Respiratory rate (RR) devices: measurement properties

Supplemental Table S3 outlines the methodological quality and measurement results for each study examining devices reporting on RR.

Reliability

No studies evaluated the reliability of devices used for measuring RR.

Convergent validity

One study evaluated the convergent validity of RR devices against other measures. Soler et al. [25] evaluated the agreement between the TeleOx ®, an oxygen flow rate monitor with a pressure sensor to capture RR, and a polygraph (Nox-T3®) in participants undergoing nasal oxygen therapy. The TeleOx ® consistently reported 0.05 more respiratory cycles per minute (95% LoA=-3.865, 3.957).

Known-groups validity

No studies evaluated the known-groups validity of devices used for measuring RR.

Concurrent validity

One study evaluated the concurrent validity of RR devices against a gold standard device. Rubio et al. [26] examined the agreement of 5 RR devices: (1) chest mounted electrode array, (2) finger mounted photoplethysmography system, (3) camera mounted distance photoplethysmography device, (4) upper abdomen mounted triaxial accelerometer and (5) chest worn pressure sensor pad, against a gold standard metabolic system device: the Oxycon Mobile®. Devices recorded RR while participants performed a set of daily living activities (e.g. sitting, walking, climbing stairs) over 57 minutes. The Bland-Altman results of these 5 devices with Oxycon were combined in a forest plot (Figure 2). The pooled mean difference was −1.73 (95% LoA=-5.92, 2.45) (i.e. the RR devices consistently reported 1.73 less breaths per minute than the Oxycon). There was no heterogeneity between the devices (I²=0.0%, p = 0.97).

Figure 2. Forest plot of the agreement between 5 respiratory rate devices [26] and the Oxycon (gold standard). Electrode: chest mounted electrode array, PPG: finger mounted photoplethysmography system, Camera: camera mounted distance photoplethysmography device, Accel: upper abdomen mounted triaxial accelerometer, Sensor-pad: chest worn pressure sensor pad. LOA: limits of agreements.

Predictive validity

One study evaluated the predictive validity of RR devices in detecting an exacerbation event. Shah et al. [27] found that RR measured by a photoplethysmogram (which was recorded by a pulse oximeter) almost met the 0.70 cutoff for acceptable accuracy when predicting an exacerbation event (AUC = 0.605).

Responsiveness

No studies evaluated the responsiveness of devices used for measuring RR.

Discussion

The COVID-19 pandemic has highlighted an urgent need for the use of remote health monitoring devices. This systematic review suggests that digital devices used for remote monitoring SpO₂ and RR may be valid in individuals with COPD and may be used by clinicians to estimate the oxygen status and breathing rate of COPD patients in remote settings. Our results demonstrated that SpO₂ and RR devices were generally valid when compared against other respiration monitoring devices; however, further evaluation is warranted for SpO₂ devices during activity as there were inconsistent findings as to how well they agreed with other devices under exercise conditions. Well-designed measurement studies concerning digital respiration monitoring devices are needed in order to make firm conclusions about the accuracy of their measurements.

International Organization for Standardization (ISO) standards are important in establishing safety and accuracy of medical devices. With respect to validity, ISO standards determine the objective accuracy of devices by calibrating them against a known gold standard. For example, pulse oximeters seek to provide a measure of the oxygenation of blood using a noninvasive technique [29]. This systematic review was performed following COSMIN guidelines for evaluating and reporting measurement properties. The COSMIN guidelines provide a detailed and comprehensive list of requirements to summarize and evaluate the evidence on the reliability and validity of health outcome measures. The COSMIN guidelines consider the extent to which measures assessing similar constructs compare, the ability of the measure to discriminate between subgroups of individuals and the sensitivity of the measure to change. This review showed that studies evaluating the concurrent validity of pulse oximeters found these devices to be valid as they demonstrated agreement with a gold standard device or criteria. These findings support the work conducted by ISO standards to ensure accuracy of medical devices.

We performed a meta-analysis to provide an overall estimate of the degree to which RR monitoring devices deviated from a gold standard measure, the Oxycon. The pooled analysis indicated that RR devices consistently reported approximately two breaths less, per minute, than the Oxycon. Although the data points included in the plot arose from the same study, which could have biased the overall pooled estimate, findings suggest that clinicians should be aware that RR monitoring devices may underestimate the RR of patients. For predictive validity, SpO₂ and RR devices examined almost met the acceptable AUC cutoff of 0.70 in predicting exacerbation events. Although promising, these results highlight the need to perform further measurement studies to confirm the validity of monitoring devices in COPD.

This review identified that there is a need to investigate the reliability, known-groups validity and responsiveness of these devices as there were only a few studies reporting on these properties. Reliability allows us to determine whether these devices provide similar outcomes overtime when COPD patients are in a stable condition [13]. If SpO₂ or RR devices are reliable, then the probability of a false alarm (i.e. an inaccurate abnormal recording of SpO₂ or RR) is reduced and the probability of a true alarm is increased. This reduces alarm fatigue [30] and allows care to be directed to where it is needed; thus, reducing unnecessary healthcare costs and improving patient care. Known-groups validity allows us to determine whether respiration monitoring devices are able to distinguish between different clinical characteristics [13], such as different airflow limitation severities defined by the Global Initiative for Chronic Obstructive Lung Disease [1]. This form of validity permits devices to provide respiration outcomes (i.e. SpO₂, RR) reflecting the severity of the disease. Responsiveness provides the opportunity to determine whether these devices can capture changes in SpO₂ or RR when there has been a change in the patient’s respiratory function (e.g. an improvement after pulmonary rehabilitation) [13]. Respiration monitoring devices that are responsive help inform clinicians and researchers of improvements and/or deteriorations of patients’ health status in response to an intervention. Overall, the evidence for these properties, in addition to convergent, concurrent and predictive validity, is needed to confirm the accuracy of these respiration devices in representing the health status of individuals with COPD and to ensure the delivery of appropriate care.

Inadequate methodological quality for the included studies was mainly attributed to the employment of inadequate statistical methods and/or poor reporting on the measurement properties of the comparator measure. For example, Burton et al. [25] used only odds ratios to evaluate predictive validity of a pulse oximeter. However, as recommended by COSMIN, also including a Receiver Operating Characteristic curve (to obtain AUCs) would have been informative in determining how accurately a SpO₂ device can predict those with exacerbations versus those without [29], as odds ratios only examine the association of exposure (e.g. decrease in SpO₂) and outcome (e.g. exacerbation event) [25, 31]. Furthermore, lack of information on the measurement properties of the comparator measure makes it difficult to interpret the findings regarding the measure under study [14]. If the comparator’s ability to accurately and consistently measure respiration is unclear, then hypotheses regarding the correlation of the device with the comparator are invalid.

Reliable and valid respiration devices enable effective monitoring of COPD patients’ health conditions; facilitating the management of symptoms and the prevention of exacerbation events. Appropriate management of symptoms and prevention of exacerbation events will reduce the healthcare costs associated with COPD (e.g. emergency room visits, medications); thus, lessening economic burden to individuals and society [32]. With the increase use of telerehabilitation [33], reliable and valid monitoring equipment is needed to ensure patients’ stability and identify adverse health events (e.g. sudden increase or decrease in oxygen/respiration). Furthermore, once the reliability and validity of respiration monitoring devices have been established in COPD, hypotheses can be made about their performance in individuals with COVID-19, as COVID-19 primarily affects the respiratory system [29]. Remote monitoring of respiration can be used to track recently discharged COVID-19 patients for declines in health status (e.g. breathing difficulties) while preserving healthcare resources [31].

Pulse oximeters have been commonly used for monitoring the status of patients’ oxygen levels in hospital settings [34]; however, their use in remote settings has recently emerged with advancements in telemonitoring, and their importance has amplified as a result of the COVID-19 pandemic [35]. Respiration monitoring devices are currently being used in telemonitoring settings, even though sufficient evidence of their performance has not been documented. Due to limited measurement evidence, no recommendations regarding which model of device to use in remote settings can currently be made. Therefore, a rigorous and exhaustive inspection of current respiration monitoring devices’ psychometric properties is urgently needed to ensure consistent and accurate SpO₂ and RR outcomes in remote settings.

Author contributions

All authors (AM, EW, JR, MB, AK) contributed to the study’s conception and/or design. Screening of articles and quality assessments were performed by AM and EW. The manuscript was written by AM and AM is the guarantor of the paper. All authors have commented on previous versions of the manuscript and approved the final manuscript.

Declaration of interest

The authors report no conflicts of interest.