Validation of ACCESS: an automated tool to support self-management of COPD exacerbations

Abstract

Background

To support patients with COPD in their self-management of symptom worsening, we developed Adaptive Computerized COPD Exacerbation Self-management Support (ACCESS), an innovative software application that provides automated treatment advice without the interference of a health care professional. Exacerbation detection is based on 12 symptom-related yes-or-no questions and the measurement of peripheral capillary oxygen saturation (SpO₂), forced expiratory volume in one second (FEV₁), and body temperature. Automated treatment advice is based on a decision model built by clinical expert panel opinion and Bayesian network modeling. The current paper describes the validity of ACCESS.

Methods

We performed secondary analyses on data from a 3-month prospective observational study in which patients with COPD registered respiratory symptoms daily on diary cards and measured SpO₂, FEV₁, and body temperature. We examined the validity of the most important treatment advice of ACCESS, ie, to contact the health care professional, against symptom- and event-based exacerbations.

Results

Fifty-four patients completed 2,928 diary cards. One or more of the different pieces of ACCESS advice were provided in 71.7% of all cases. We identified 115 symptom-based exacerbations. Cross-tabulation showed a sensitivity of 97.4% (95% CI 92.0–99.3), specificity of 65.6% (95% CI 63.5–67.6), and positive and negative predictive value of 13.4% (95% CI 11.2–15.9) and 99.8% (95% CI 99.3–99.9), respectively, for ACCESS’ advice to contact a health care professional in case of an exacerbation.

Conclusion

In many cases (71.7%), ACCESS gave at least one self-management advice to lower symptom burden, showing that ACCES provides self-management support for both day-to-day symptom variations and exacerbations. High sensitivity shows that if there is an exacerbation, ACCESS will advise patients to contact a health care professional. The high negative predictive value leads us to conclude that when ACCES does not provide the advice to contact a health care professional, the risk of an exacerbation is very low. Thus, ACCESS can safely be used in patients with COPD to support self-management in case of an exacerbation.

Keywords:

• COPD
• exacerbations
• telehealth
• software application
• treatment advice
• self-management
• health
• mobile health
• automated device
• diagnostic accuracy

Acknowledgments

We would like to thank all patients who participated in this study for their commitment to the study, as well as all team members, especially Marleen Kolenbrander and Samantha van der Hoogen, the nurses, pulmonary nurses, and pulmonologists, of the rehabilitation department of Dekkerswald, and Heleen van der Niet and Netty Plat, pulmonary nurses of the outpatient clinic of the Radboudumc. Furthermore, we would like to thank Jeanine Antons, MD, for her contribution to the design of the present study, and Professor Yvonne Heijdra, MD, Dr Johan Molema, MD, and all the health care professionals of the expert panel for their contribution to the development of ACCESS. This study was funded by the Netherlands Organization for Health Research and Development, Boehringer Ingelheim, and the Radboud University Medical Center. None of the funding bodies had any role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Disclosure

None of the authors received any support from any company for the submitted work. All authors have completed the Unified Competing Interest form at http://www.icmje. org/conflicts-of-interest/ (available on request from the corresponding author). PL has a patent 20140206949 issued to Petrus Lucas; EB received personal fees from Boehringer Ingelheim and GlaxoSmithKline, outside the submitted work; TS received a grant from GlaxoSmithKline and personal fees from Boehringer Ingelheim, outside the submitted work. The other authors report no conflicts of interest in this work.

Author contributions

All authors contributed to data analysis, drafting and revising the article, gave final approval of the version to be published, and agree to be accountable for all aspects of the work.

Supplementary materials

Previous steps in the development of ACCESS

The aim of the ACCESS system is to support COPD patients in exacerbation self-management by detecting exacerbations and providing tailored and timely treatment advice.

Building and optimizing the Bayesian network model

van der Heijden et al¹ have described the first develop-mental phase of ACCESS (Figures S1 and S2, Step 1). In summary, a Bayesian network model was constructed based on expert opinion of two pulmonary physicians, who pro-vided relevant parameters based on their clinical experience and contemporary literature. A Bayesian network model is a flexible probabilistic model that can compute the probability of the presence of a disease based on available symptoms, also in case of missing data, and predictions can be personalized by entering patient-specific data. Based on the expert-driven model and available data, a data-driven model was created and compared to the occurrence of symptom-based exacerbations.¹ This resulted in a risk prediction of exacerbations based on 12 yes or no questions which are complemented with measurements of peripheral capillary oxygen saturation (SpO₂), forced expiratory volume in one second (FEV₁), and body temperature. Furthermore, a pilot study among patients with COPD was performed to examine usability of the smart phone and sensors.¹

Primary aim of the study described in the current paper was to gather data for a cross-validation procedure to optimize and test the Bayesian network model to automatically detect exacerbations. The data were divided in six groups of nine randomly selected patients. In each of the six cross-validity checks, the model learned to identify exacerbations based on the entries of the five training sets and was then tested for diagnostic accuracy on the sixth set. Gold standard were symptom-based exacerbations. The final model, resulting from the cross-validation procedure, showed an area under curve of the receiver operating curve of 91.4% (95% CI 90.3–92.5; Figure S3).

After the cross-validation procedure, a cutoff point of 85% was chosen to indicate when an exacerbation was at hand. Any risk calculation beyond this point should lead to an automatic advice of ACCESS to contact a health care professional or start a course of prednisolone.

Development and integration of automatic treatment advices

The next goal was to formulate treatment advice based on the risk calculation of the Bayesian network model to be given automatically by ACCESS. However, when discussing the matter in detail with clinicians, it became clear that self-management advice could not solely be based on the risk calculation of ACCESS. To decide in which specific situation what advice should be given by ACCESS, clinical expertise was used (Figure S2, Step 2). Expert involvement was realized in a three-step procedure: 1) an expert panel consisting of a pulmonary physician, a general practitioner, and a clinical psychologist discussed with which symptoms they wanted a patient to take what action, based on their clinical experience and contemporary guidelines for COPD exacerbation management; 2) various health care professionals – pulmonary physicians, pulmonary nurses, general practitioners, practice nurses – provided the advices they would give in four standardized cases on paper; and 3) these advices were then discussed by the expert panel and minor adjustments were made to the decision model.

In contrast to what we had previously anticipated, the expert panel focused on its clinical judgment, rather than using the risk calculation of the Bayesian network model as a base for the advices. This was perhaps inevitable in many cases, since several advices are initiated based on specific symptoms. Eventually, a cutoff point of the risk calculation of the Bayesian network model was chosen to underlie the expert panel’s decision model for automatic advice. This cutoff point was set at 85%, with a sensitivity of 65.8% (95% CI 62.4–69.0), specificity of 91.5% (95% CI 90.2–92.6), positive predictive value of 75.0% (95% CI 71.6–78.1), and negative predictive value of 87.3% (95% CI 85.8–88.6). When this cutoff point is reached, ACCESS will always provide the advice to contact a health care professional.

The advices are personalized by the patient’s health care professional, who enters the patient’s FEV₁ and SpO₂ values in the system as reference values, and adds specific medication instructions for maximal bronchodilator use. Specific advices can be blocked for a patient, if his health care professional deems this best. For instance, if the patient has not been taught coughing techniques, the advice to use coughing techniques would be irrelevant, maybe even confusing.

As a last step in the developmental phase, we have examined in the current study if patients are advised safely and correctly, by comparing the clinically most important ACCESS advice – “Contact a health care professional today” – with a symptom-based and an event-based definition of exacerbations (Step 4). For a clinical evaluation of ACCESS, a randomized controlled trial is in progress, comparing the effect of ACCESS with the effect of a written action plan on exacerbation time (Step 5).

Figure S1 Schematic presentation of the ACCESS system.

Abbreviation: ACCESS, Adaptive Computerized COPD Exacerbation Self-management Support.

Figure S2 Schematic overview of the development of the Adaptive Computerized COPD Exacerbation Self-management Support (ACCESS) System.

Figure S3 ROC curve of cutoff values of the Bayesian network model’s risk calculation to identify a symptom-based exacerbation.

Abbreviations: ROC, receiver operating curve; AUC, area under the curve.

Reference

• van der HeijdenMLucasPJLijnseBHeijdraYFSchermerTRAn autonomous mobile system for the management of COPDJ Biomed Inform201346345846923500485
   PubMed Web of Science ®Google Scholar