Publication Cover
Canadian Journal of Respiratory, Critical Care, and Sleep Medicine
Revue canadienne des soins respiratoires et critiques et de la médecine du sommeil
Volume 2, 2018 - Issue 4
1,704
Views
3
CrossRef citations to date
0
Altmetric
CTS Guidelines and Position Papers

Pan-Canadian asthma and COPD standards for electronic health records: A Canadian Thoracic Society Expert Working Group Report

, , , , , , , , , , , , , , & show all
Pages 244-250 | Received 20 Aug 2018, Accepted 24 Aug 2018, Published online: 05 Nov 2018

Abstract

RATIONALE: The Canadian Thoracic Society established a pan-Canadian respiratory standards initiative for electronic health records (EHRs) (PRESTINE).

OBJECTIVE: We aimed to identify and define respiratory data elements for EHRs for asthma, COPD and related pulmonary function elements that enable adherence with respiratory best practice guidelines.

METHOD: Potential data elements (n = 425) were based on a published asthma/COPD information model. Using modified RAND-UCLA Appropriateness and Delphi methods, a working group (WG) of 12 experts independently rated each element based on 4 domains (strength of evidence, clarity, relevance, feasibility) using a 5-point Likert Scale, plus an overall rating (include as core, optional or exclude). Subsequent independent voting rounds addressed elements lacking consensus (defined as 60% agreement) in previous rounds. A facilitated face-to-face meeting was convened during which WG consensus was sought. A list of included data element definitions and medications were sent for external stakeholder review.

MAIN RESULTS: After 4 rounds of voting (including the face-to-face meeting), the WG identified 77 core and 23 optional elements for asthma, and 72 core and 21 optional elements for COPD. Of those, 53 core and 15 optional elements were common to both asthma and COPD. The list of asthma/COPD and smoking cessation medications included 40 products and 48 brands.

CONCLUSIONS: This consensus initiative has identified asthma, COPD, and pulmonary function data elements and definitions as well as a list of medications recommended by experts for inclusion in EHRs to support primary and tertiary care for these diseases, and to enable outcomes monitoring, benchmarking and performance evaluation.

RÉSUMÉ

JUSTIFICATION: La Société canadienne de thoracologie a mis sur pied une initiative pancanadienne sur les normes respiratoires pour les dossiers de santé électroniques (PRESTINE).

OBJECTIF: Nous avions pour but de déterminer et de définir les éléments de données respiratoires à inclure dans les dossiers de santé électroniques pour l’asthme, la MPOC et les éléments de la fonction pulmonaire afférents qui facilitent l’adhésion aux lignes directrices pour de meilleures pratiques respiratoires.

MÉTHODE: Les éléments de données potentiels (n = 425) étaient fondés sur un modèle d’information publiée sur l’asthme ou le MPOC. À l’aides des méthodes Delphi et RAND-UCLA Appropriateness, un groupe de travail de 12 experts a noté chaque élément de manière indépendante relativement à quatre domaines (force des données probantes, clarté, pertinence et faisabilité) en utilisant une échelle de Likert en 5 points, à laquelle s’est ajoutée une note globale (inclure comme élément central, inclure comme élément optionnel ou exclure). Les tours de vote indépendants subséquents ont porté sur les éléments qui n’avaient pas fait consensus (défini comme un accord à 60 %) dans les tours antérieurs. Une rencontre face à face avec facilitateur a été tenue en vue d’atteindre un consensus du groupe de travail. Une liste des définitions des médicaments et des éléments de données inclus a été envoyée à des parties prenantes externes pour examen.

PRINCIPAUX RÉSULTATS: Après quatre tours de vote (y compris la rencontre face à face), le groupe de travail a répertorié 77 éléments centraux et 23 éléments optionnels pour l’asthme, ainsi que 72 éléments centraux et 21 éléments optionnels pour la MPOC. Parmi ceux-ci, 53 éléments centraux et 15 éléments optionnels étaient communs à l’asthme et à la MPOC. La liste des médicaments pour l’asthme, la MPOC et la cessation tabagique comprenait 40 produits et 48 marques.

CONCLUSIONS: Cette initiative de consensus a répertorié les éléments de données et les définitions en matière d’asthme, de MPOC et de fonction pulmonaire, ainsi qu’une liste des médicaments recommandés par les experts, devant être inclus dans les dossiers médicaux électroniques afin de soutenir les soins primaires et tertiaires pour ces maladies, ainsi que pour permettre le suivi des résultats, le benchmarking et l’évaluation de la performance.

Introduction

Chronic respiratory disease affects over 3 million Canadians, and excluding cancer is the third leading cause of death from chronic disease in Canada.Citation1 Despite publication of national evidence-based guidelines for the diagnosis and management of asthmaCitation2,Citation3 and COPD,Citation4,Citation5 care gaps remain and the importance of knowledge translation and implementation initiatives is increasingly recognized.Citation6,Citation7 Performance measurement, benchmarking and continuous quality improvement have been identified as provincial and national health system priorities; yet, to date, practical systems that support chronic disease management are not routinely available. Electronic health records (EHRs) are increasingly prevalent, particularly in primary care, and represent a novel opportunity to integrate guidelines into day-to-day practice.Citation8

Recognizing the need for electronic medical records (EMRs), the Government of Ontario funded a pilot project to determine the ability to incorporate asthma data elements from the The Lung Association – Ontario’s (TLA’s) Asthma Care Map (ACM) (a paper form used for clinical documentation in Ontario’s Primary Care Asthma Program sites) into an electronic format, and to send de-identified data to a central server for analysis and report generation. Cross-referencing of the ACM data elements to SNOMED® (Systemized Nomenclature for Medical Clinical Terms) and LOINC® (Logistical Observation Identifier Names and Codes) revealed a high level of congruency (100% of laboratory variables and the majority of clinical variables were exact [47.8%] or partial [17.8%] matches).Citation9 Two “stand-alone” asthma EMR systems [the Kingston Health Sciences Centre Asthma Program’s Asthma Management and Outcomes Monitoring System (AMOMS) (integrated into the hospital’s patient care system) and Windsor’s Asthma Research Group, Incorporated (ARGI)] succeeded in incorporating 69 asthma data elements and data definitions established by the clinician researchers for the pilot project.Citation6 A participating EHR vendor (approved by Ontario’s EHR licensing body, OntarioMD, as a certified vendor) was unable to do so within the time frame of the project.Citation6 Reporting was feasible, but the analysis was time-consuming, and many challenges were noted, including missing data and variable programmer interpretation of the data definitions. In 2008, asthma data elements and definitions developed in this pilot project that related to follow-up visits were endorsed by OntarioMD, as minimum specifications for approved primary care her vendors.

Given existing literature on asthma documentation in EMRs in primary care and our experience with the above pilot project, we recognized the need and potential for standardized respiratory disease terminology and nomenclature for EMRs, to facilitate performance evaluation, benchmarking and continuous quality improvement.Citation6

Recognizing that EHR data standards are of national relevance, the Canadian Thoracic Society (CTS), in collaboration with TLA, launched a Pan-Canadian REspiratory STandards INitiative for Electronic Health Records (PRESTINE) initiative.Citation8 Stakeholders attended a national workshop in October 2011 to develop a strategy to recommend respiratory data elements and standards for use in electronic medical records across Canada that meet the needs of providers, administrators, researchers and policy makers, in order to facilitate evidence-based clinical care, monitoring, surveillance, benchmarking and policy development. Workshop attendees recommended the project focus initially on asthma and related data elements for COPD, smoking history and pulmonary function elements that are applicable to many respiratory conditions.

The objective of this initiative was to identify evidence-based respiratory data elements and standards for asthma (and related COPD and pulmonary function elements) for use in EHRs across Canada.

Methods

Modified RAND-UCLA Appropriateness and Delphi methods were utilized to achieve consensus on data elements.Citation10,Citation11 Ethics approval for the study was obtained from the Queen’s University and Affiliated Teaching Hospitals’ Health Sciences Research Ethics Board.

Expert working group

A Working Group (WG) panel of 12 experts in the areas of adult and pediatric respirology focusing on asthma, COPD, pulmonary function testing, and health services/population health research from across Canada was assembled. The panel consisted of: 7 physicians (including 5 respirologists, 1 pediatrician and 1 family physician), 3 Registered Respiratory Therapists & Certified Respiratory Educators, 1 population health research scientist and 1 pulmonary function standards expert. Key stakeholders were also identified to be included in the consultation process.

Data element selection

A comprehensive list of potential data elements and indicators was compiled from several sources, including: an Asthma/COPD information model devised by M. Jurlink, based upon review of the literature and an environmental scan completed within the TLA’s Asthma EMR Project;Citation8 the Ontario Tobacco Research Unit’s Smoking Cessation Data Standards;Citation12 published Primary Care Asthma Performance Indicators (PC-API©);Citation13 COPD Performance IndicatorsCitation14 and input from the WG experts. Consensus on medications to be included in the guidelines was reached separately from the Delphi voting rounds. A list of medications was distributed to the WG members for review and feedback.

Modified Delphi process and face-to-face consensus meeting

Using a modified Delphi process,Citation15 4 rounds of voting were completed. In the first round, due to the large number of potential elements, the WG was divided into sub-groups of 3–4 panelists. Each of the four sub-groups completed independent assessments of ∼25% of the total potential data elements. Experts independently rated each element based on 4 domains (strength of evidence, clarity, relevance, feasibility) on a 5-point Likert scale) as well as an overall ranking (include as core, include as optional, or do not include) (). All potential data elements were categorized as core, optional or exclude for Asthma and COPD independently. Data elements categorized as core or optional were included for Asthma, COPD or both (if they happened to overlap). Consensus was defined as ≥60% agreement among the voting panel members. Elements that did not reach consensus were deemed to be contentious and moved to the subsequent round for further discussion and voting. Round 1 ratings were discussed among sub-groups by teleconference. In each subsequent round, all WG members voted on all contentious elements from the previous round.

Figure 1. Rating scheme used in Rounds 1 and 2 voting. Ratings were based on 4 domains (strength of evidence, clarity, relevance and feasibility) using a 5-point Likert scale, as well as an overall ranking to include as core, include as optional or do not include for asthma, COPD or both. An example of the results for the category of “Irritant Triggers” is shown.

Figure 1. Rating scheme used in Rounds 1 and 2 voting. Ratings were based on 4 domains (strength of evidence, clarity, relevance and feasibility) using a 5-point Likert scale, as well as an overall ranking to include as core, include as optional or do not include for asthma, COPD or both. An example of the results for the category of “Irritant Triggers” is shown.

In Round 3, a face-to-face facilitated consensus meeting was convened, led by an expert group facilitator from Queen’s University Executive Decision Center at the Smith School of Business. Ten of 12 WG members attended in person, and the remaining 2 members attended via teleconference. All members voted using an electronic voting system. Panelists openly discussed all selected data elements for group consensus and reviewed short-listed elements, followed by a round of voting. Remaining contentious elements from the meeting were voted on in Round 4 by all WG members.

Following categorization of all data elements, a comprehensive list of the included elements and definitions was distributed to the WG members for feedback and editing prior to the external stakeholder review. Definitions were discussed and edited for final consensus. Definitions were cross-referenced to standardized nomenclature in SNOMED© and LOINC©, and exact matches were indicated to the WG prior to voting, with the recommendation that those definitions not be altered without compelling clinical rationale. The research team reviewed the final list of elements and definitions, removed duplicate elements that had originated from different care maps, reorganized the element list such that response options were counted as sub-elements and revised definitions to be internally consistent in format.

External stakeholder review

For the external review, 11 stakeholders were invited to participate with expertise in eHealth, data standards and terminology, health research, primary health information and population health, including the following organizations: Canada Health Infoway, the Canadian Institute for Health Information (CIHI), the Public Health Agency of Canada (PHAC), and The Lung Association – Ontario (TLA). The review by Canada Health Infoway was limited to a data standards perspective, and was not done from clinical content or business requirements perspectives. Stakeholder participants independently completed an electronic survey to review all data elements and definitions, and propose revisions as needed. Duplicate elements were removed and formatting of definitions was revised to improve consistency. Participants were also asked to comment on the content and format of the draft manuscript, as well as the overall coherence and comprehensiveness of the element list.

Results

A total of 425 data elements within 28 categories were selected for the Delphi Panel review. The results of each round of voting on data elements are summarized in . After Round 1, 225 elements were voted as core, 67 as optional and 38 data elements were excluded. A list of 95 data elements remained contentious and was moved to Round 2 for further voting.

Figure 2. Flow chart of the initial data element selection process. The Working Group rated 425 potential elements in 4 rounds of voting as core, optional or to be excluded. Elements lacking consensus in a given round were deemed “contentious” and addressed in the next round. Subsequently, duplicate elements were removed. *The sub-totals displayed for asthma and COPD include data elements that are common to both. The element list was further revised based upon data definition review, which included consideration of standardized terminology and reorganization of data element response options into sub-elements.

Figure 2. Flow chart of the initial data element selection process. The Working Group rated 425 potential elements in 4 rounds of voting as core, optional or to be excluded. Elements lacking consensus in a given round were deemed “contentious” and addressed in the next round. Subsequently, duplicate elements were removed. *The sub-totals displayed for asthma and COPD include data elements that are common to both. The element list was further revised based upon data definition review, which included consideration of standardized terminology and reorganization of data element response options into sub-elements.

After Round 2, 33 data elements were removed for redundancy, and 57 remained contentious. In Round 3, following voting on contentious elements, a round of indicator voting addressed all data elements at the facilitated consensus meeting. From Round 3 and onward, elements were specified to be included, optional, or excluded for Asthma, COPD or both.

After 4 rounds of voting, a sub-total of 266 data elements had been identified for data definition review. Of this sub-total, 83 elements were voted as core for Asthma, 27 were voted as core for COPD and another 97 elements were voted as core for both Asthma and COPD. A total of 22 elements were voted as optional for Asthma, 27 as optional for COPD, and an additional 20 elements were voted as optional for both Asthma and COPD.

Next, data elements and definitions were reviewed by the WG and the research team for redundancy as well as consistency of formatting (for example for frequency variables). Duplicate elements were removed. Elements determined to be ‘response options’ for other elements were more appropriately categorized as ‘sub-elements.’

WG members added three medications to the list circulated by the research team. The final list of medications recommended for inclusion included 12 medication classifications, with a total of 38 different products and 64 brands. The number of medications by products and brands by medication class are tabulated in .

Table 1. Medications by product and brand.

External stakeholder review

A total of 9 of the 11 invited external stakeholders reviewed the final data element and medication lists. One stakeholder sought additional input from 4 colleagues within their organization (CIHI). Stakeholder feedback was reviewed by the research team. Data definitions and formatting of elements and sub-elements was revised accordingly to improve consistency. Four additional duplicate elements were removed, 45 elements/sub-elements were added to make the element list more comprehensive for COPD, and 2 new medications were added.

The final number of core and optional elements and sub-elements recommended for Asthma, COPD and both Asthma and COPD are illustrated in . The final list of elements, sub-elements and definitions/permissible values are listed in Appendix A. The final list of medications are listed in Appendix B.

Figure 3. Venn diagram illustrating the final number of A) Core and B) Optional elements for asthma, COPD and both Asthma and COPD. The number of response options to data elements (categorized as “sub-elements”) are shown in brackets.

Figure 3. Venn diagram illustrating the final number of A) Core and B) Optional elements for asthma, COPD and both Asthma and COPD. The number of response options to data elements (categorized as “sub-elements”) are shown in brackets.

Discussion

Using combined modified RAND-UCLA Appropriateness and Delphi methods, this initiative has identified a list of core and optional elements that support evidence-based care of individuals with asthma and COPD, deemed by experts to merit inclusion in primary care EHRs/EMRs. Many elements, such as how pulmonary function tests are documented, were determined to be common to both diseases. Asthma, COPD and smoking cessation medications were also identified. Definitions for the elements and sub-elements have been proposed, including identification of established standardized nomenclature/terminology in SNOMED© and LOINC©.

EHRs and EMRs are increasingly prevalent, particularly in primary care, and represent a novel opportunity to integrate guidelines into day-to-day practice.Citation8 The potential benefits of standardization of data elements for EHRs and EMRs include improved quality of care and access to care, increased productivity, and facilitation of outcomes monitoring, benchmarking and performance measurement.Citation8,Citation13 EHRs supported by defined, coded data elements also enable EHR system interoperability.Citation8 To achieve these goals, there is a need for consistent collection of high quality data, with minimal use of scanned documents or free-text data entry.Citation16

In 2011, the Canadian Institute for Health Information (CIHI) released a Draft Pan-Canadian Primary Health Care Electronic Medical Record Content Standard (PHC EMR CS).Citation17 It included 106 core data elements, designed to meet the expressed need of clinicians and health system administrators for better and consistent data for patient care, outcomes monitoring, and health policy development. To promote adoption, a priority subset of these elements was identified in 2014 (Version 3.0). The PRESTINE results align with this approach, focusing on the two most common chronic respiratory diseases.

By including both asthma and COPD elements in our initial list for voting, data elements common to both conditions were able to be identified. Many data elements were deemed relevant for both asthma and COPD, and some, such as smoking history, have even broader relevance. While much of the focus was on core elements for inclusion in primary care EMRs, the elements and definitions are relevant across practice settings, including specialist clinics and respiratory educator practices. Pulmonary function tests and asthma, COPD and smoking cessation medications have also been identified for potential inclusion in EMRs. Recognizing that Canada Health Infoway is developing a drug terminology called the Canadian Clinical Drug DatasetCitation18 for use in digital health solutions such as prescribing, there may be opportunities to align our findings with this initiative.

Similar to the potential benefits of implementation of the PHC EMR CS and Canadian Clinical Drug Dataset, adoption of the PRESTINE elements has the potential to greatly enhance the utility of EMRs and EMR data for clinical care and research, and complement national chronic disease surveillance systems. For example, the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) has validated a case-definition and case-finding algorithm for COPD.Citation19 Integration of PRESTINE COPD elements into EMRs would likely facilitate analysis of COPD EMRCitation20 data in CPCSSN extractions.

Although terminology/nomenclature was initially deemed beyond the scope of this initiative, we have collaborated with data standard experts from Canada Health Infoway and eHealth Ontario, to augment earlier ACM cross-referencing work and identified PRESTINE data elements for which existing SNOMED© and/or LOINC© standards exist.Citation9 While post-coordination in SNOMED© (a compositional process that combines two or more concepts to provide clarity and explicitly to clinical data) can improve mapping results, it relies on adherence to descriptive logic rules that are complex and not intuitive.Citation9 As such, we did not attempt it but plan to collaborate with experts from Canada Health Infoway in the future to examine the use of pre- and post-coordination, as well as alignment with other health terminologies. Furthermore, through Canada Health Infoway, we will request new concepts and changes to SNOMED CT© Canadian Edition and LOINC© for core data elements not matched.

A major strength of this initiative is the use of a diverse interdisciplinary group of experts for the Delphi panel, and inclusion of an external review process that involved specialty societies, guidelines panels, as well as data standards and eHealth experts. Additionally, evidence-based asthma and COPD guidelines and care maps were used to inform and guide data element selection and voting. The process established by this initiative may be useful for other organizations wishing to achieve expert consensus on EHR elements for other diseases.

The main limitations of our findings are those inherent to research methods of Delphi panels and consensus groups.Citation21 It is possible that the experts who participated in the panel brought biases to their voting, and in the face-to-face meeting lack of anonymity may have affected the discussion. Use of an expert group facilitator who was not a clinician or content expert to guide group discussion should minimize this risk. Similarly, the researchers may have introduced bias inadvertently by selecting data elements primarily from the asthma and COPD tools with which they are most familiar. There are still many barriers to effective EMR adoption and use of EMR data by stakeholders, not the least of which are interoperability, data privacy and data sharing.Citation22

In conclusion, this project engaged asthma, COPD and pulmonary function standards clinical and research experts, and used an evidence-based approach to identify data elements and definitions recommended to be included in EHRs. Many elements were deemed core and optional for both asthma and COPD, reducing the total number of elements overall. This standardization process has established an approach to defining elements for EHRs that support best practice, which may be replicated for other chronic diseases. Implementation of these elements and standards will enable outcomes monitoring, EHR interoperability, benchmarking and performance evaluation.

Supplemental material

Supplemental Material

Download Zip (1 MB)

Acknowledgments

The authors wish to thank Erik Lockhart, Associate Director, Executive Decision Centre, Smith School of Business, Queen’s University, for his expert facilitation of the face-to-face working group meeting.

The authors are also grateful to the external stakeholder reviewers for their valuable input: Sharon Bartholomew, Health Research Scientist, PHAC; Richard Birtwhistle, eHealth Expert, Queen’s University; Louis-Philippe Boulet, Knowledge Translation Chair in Respiratory and Cardio-Vascular Health, Laval University; Mary Byrnes, Manager, Primary Health Care Information, CIHI; Keith Denny, Director, Clinical Data Standards and Quality, CIHI; Sara Grimwood, Program Lead, Population Health, CIHI; Geoff Hynes, Manager, Population Health, CIHI; Andrea MacLean, Data Standards/Terminology Expert, Canada Health Infoway; Raymond Simkus, Data Standards/Terminology Expert, Brookswood Family Practice; Janice Spence, Data Standards/Terminology Expert; Kathleen Spurr, Registered Respiratory Therapist, Dalhousie University; Marie Tran, Senior Analyst, Primary Health Care Information, CIHI.

The opinions, results and conclusions reported are those of the authors and are independent from the provincial, and territorial governments of Canada.

Disclosure statement

M. Diane Lougheed has received grants outside the submitted work paid directly to Queen’s University from the Ontario Lung Association, Ontario Thoracic Society, the Government of Ontario’s Innovation Fund, AllerGen NCE, Queen's University, GlaxoSmithKline, Hoffman LaRoche, Janssen, and Novartis, honoraria from the Ontario Lung Association for preparation and review of educational materials, and honoraria from AstraZeneca for participation in the Precision Program Advisory Board.

Ann Taite, Alison Morra, and Anne van Dam have no disclosures to report.

Francine Ducharme has received grants outside of the submitted work paid to the Sainte-Justine University Health Centre Research Institute from the Canadian Institute of Health Research, unrestricted donations from Astra Zeneca, Boehringer Ingelheim, Merck Canada, GlaxoSmithKline, and Novartis, research grants from Merck and GlaxoSmith Kline, and is serving on advisory boards for Boehringer Ingelheim, Sanofi, and Teva.

Madonna Ferrone has no disclosures to report.

Andrea Gershon has received grants through CIHR and the Government of Ontario.

Donna Goodridge has no disclosures to report.

Brian Graham has received honoraria from the Ontario Lung Association, the Lung Association of Saskatchewan and the Community Respiratory Care Program, Prince Albert Cooperative Health Centre for providing review of educational materials, teaching spirometry courses and providing in-service training for pulmonary function lab staff.

Samir Gupta has received grants and honoraria from the Ontario Lung Association.

Christopher Licskai has received grants and personal fees from AstraZeneca, grants and personal fees from Boehringer Ingelheim, grants and personal fees from Novartis, and grants from Pfizer and Bayer, outside the submitted work.

Ana MacPherson has received grants outside of submitted work paid directly to the Trillium Hospital Paediatric Asthma Centre from the RNAO Best Practice Guidelines, and to the Ontario Lung Association, Primary Care Asthma Program from the Federal Public Health Agency Canada.

Gemma Styling has no disclosures to report.

Itamar E. Tamari has received honoraria from AstraZeneca for the development, and the review of educational materials and honoraria from The Lung Association for the preparation and presentation of educational materials.

Teresa To has received funding from the Ontario Ministry of Health and Long-Term Care (MOHLTC) for the Ontario Asthma Surveillance Information System (OASIS); funding from the CIHR to link OASIS data with the CHILD study (Canadian Healthy Infant Longitudinal Development); and funding from the Ministry of the Environment and Climate Change to study air pollution and health outcomes.

Additional information

Funding

Funding was provided by the William M. Spear Endowment/Start Memorial Fund of Queen's University, the Canadian Thoracic Society, the Canadian Lung Association and the National Lung Health Framework.

References

  • The 10 leading causes of death. 2011. https://www150.statcan.gc.ca/n1/pub/82-625-x/2014001/article/11896-eng.htm.
  • Lougheed MD, Lemiere C, Ducharme FM. Canadian Thoracic Society 2012 guideline update: diagnosis and management of asthma in preschoolers, children and adults. Can Respir J. 2012;19(2):127–164.
  • FitzGerald JM, Lemiere C, Lougheed MD, et al. Recognition and management of severe asthma: A Canadian Thoracic Society Position Statement. Canadian Journal of Respiratory, Critical Care, and Sleep Medicine. 2017;1(4):199–221.
  • Bourbeau J, Bhutani M, Hernandez P, et al. CTS position statement: Pharmacotherapy in patients with COPD—An update. Canadian Journal of Respiratory, Critical Care, and Sleep Medicine. 2017;1(4):222–241.
  • O’Donnell DE, Aaron S, Bourbeau J, et al. Canadian Thoracic Society recommendations for management of chronic obstructive pulmonary disease - 2007 update. Can Respir J. 2007;14 (Suppl B):5B–32B.
  • Boulet LP, Becker A, Bowie D, et al. Implementing practice guidelines: a workshop on guidelines dissemination and implementation with a focus on asthma and COPD. Can Respir J. 2006;13 (Suppl A):5A–47.
  • Gupta S, Bhattacharyya OK, Brouwers MC, et al. Canadian Thoracic Society: Presenting a new process for clinical practice guideline production. Can Respir J. 2009;16(6):e62–e68.
  • Lougheed MD, Minard J, Dworkin S, et al. Pan-Canadian REspiratory STandards INitiative for Electronic Health Records (PRESTINE): 2011 national forum proceedings. Can Respir J. 2012;19(2):117–126.
  • Lougheed MD, Thomas NJ, Wasilewski NV, Morra AH, Minard JP. Use of SNOMED CT® and LOINC® to standardize terminology for primary care asthma electronic health records. J Asthma. 2017;55(6):629–639.
  • Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74(9):979–983.
  • Nair R, Aggarwal R, Khanna D. Methods of formal consensus in classification/diagnostic criteria and guideline development. Semin Arthritis Rheum. 2011;41(2):95–105.
  • Diemert L, Keller-Olaman S, Schwartz R, O’Connor S, Babayan A. Data standards for smoke-free Ontario smoking cessation service providers: Core indicators and questions for intake and follow-up of adult respondents; August 2013.
  • To T, Guttmann A, Lougheed MD, et al. Evidence-based performance indicators of primary care for asthma: a modified RAND Appropriateness Method. Int J Qual Health Care. 2010;22(6):476–485.
  • Gershon AS, Mecredy GC, Aaron SD, Camp PG, Tu K, Hernandez P, To T. Development of quality indicators for Chronic Obstructive Pulmonary Disease (COPD): A modified RAND appropriateness method. Can J Respir Critic Care Sleep Med. 2018. doi: 10.1080/24745332.2018.1476030. [Epub ahead of print]
  • Dalkey N, Helmer O. An experimental application of the Delphi Method to the use of experts. Manage Sci. 1963;9(3):458–557.
  • Birtwhistle R, Williamson T. Primary care electronic medical records: a new data source for research in Canada. Cmaj. 2015;187(4):239–240.
  • Sullivan-Taylor PFT, Harrison T, Webster W. Development of a draft Pan-Canadian primary health care electronic medical record content standard. International Perspectives in Health Informatics 2011;164:385–391.
  • Infoway CH. Canadian clinical drug data set. 2018; https://infocentral.infoway-inforoute.ca/en/standards/canadian/ccdd
  • Williamson T, Green ME, Birtwhistle R, et al. Validating the 8 CPCSSN case definitions for chronic disease surveillance in a primary care database of electronic health records. Ann Fam Med. 2014;12(4):367–372.
  • Lee TTK, Wing L, Gershon AS. Identifying individuals with physician-diagnosed chronic obstructive pulmonary disease in primary care electronic medical records: A retrospective chart abstraction study. NPJ Prim Care Respir Med. 2017;27:34.
  • R’Avella J. Delphi panels: Research design, procedures, advantages, and challenges. IJDS. 2016;11:305–321.
  • Terry AL, Stewart M, Fortin M. Gaps in primary healthcare electronic medical record research and knowledge: Findings of a pan-Canadian study. Healthcare Policy. 2014;10(1):46–59.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.