Validation of asthma recording in electronic health records: a systematic review

Figures & data

Figure 1 Study screening process: PRISMA flow diagram.

Note: Reproduced from Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic Reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.³⁷
Abbreviation: EHR, electronic health record.

Table 1 Characteristics of studies with validated asthma algorithms

Author, year, country, (period)	Data source, population	Sample/case characteristics	Clinical event	Algorithm	Validation
Manual validation
Xi et al,^Citation19 2015 Canada	2 large academic primary care clinics Primary care	398 randomly selected patients 16 years and older	Asthma code COPD code Other respiratory condition code Other condition code	Search algorithms: 1. Asthma in disease registry 2. Billing code 3. Asthma in CPP 4. Asthma medications 5. Asthma in chart notes 6. Asthma in CPP OR billing code 493 7. Asthma in CPP OR billing code 493 (exclusion codes 491,492, and 496) 8. (Asthma in chart notes OR asthma medications) AND billing code 493 9. (Billing code 493 OR medications) AND asthma in chart notes 10. Billing diagnostic code 493 AND asthma in chart notes	Manual review
Engelkes et al,^Citation20 2014 the Netherlands	ICPI: Dutch GP EHR Primary care	63,518 potential cases identified 22,699 cases after automated text validation Children aged 5–18	Definite, probable, and doubtful cases of asthma	Combination of ICPI communication codes, clinician codes, drug names and free text generated by a machine-learning algorithm (RIPPER)	22,699 cases manually validated, 14,303 asthma cases found
Afzal et al,^Citation21 2013 the Netherlands January 2000–January 2012	ICPI: Dutch GP EHR Primary care	63,618 potential asthma cases identified, children aged 5–18	Definite, probable, and doubtful cases of asthma	Combination of ICPI communication codes, clinician codes, drug names and free text generated by a machine-learning algorithm (RIPPER)	5,032 patients manually validated by clinician
Dexheimer et al,^Citation22 2013 United States	1 pediatric ED	15,163 assessed, 1,100 asthma patients all asthma patients (2–18 years) in a 3 month time window	Asthma code	Bayesian network system, previously used on claims data (Sanders)	Pediatric asthma/respiratory distress protocol filled in for identified patients
Wu et al,^Citation23 2013, 2014 United States	Children enrolled in the Mayo Clinic sick-child daycare program, Secondary care	112 children younger than 4	ICD-9 codes Natural language	Natural language processing (logic) Natural language processing (machine learning)	Manual review by a clinician
Kozyrskyj et al,^Citation24 2009 Canada	SAGE: birth cohort of 16,320 children born in 1995 in Manitoba, Canada Questionnaire in 2002 had 3,598 responses Manitoba’s health care registry records	723 children from the group with completed questionnaires 246 cases, 477 controls	Asthma	Database definitions in health care records	Pediatric allergist diagnosis of asthma
Pacheco et al,^Citation25 2009 United States	NUgene Project Genome-wide association study	7,970 people with DNA samples, of which 521 had an asthma diagnosis	Asthma diagnosis	Initial asthma cases algorithm: Asthma diagnosis and asthma medication prescription on ≥1 visit AND no other chronic lung disease diagnosis on ≥2 visits AND no reported smoking history ≥10 years Final asthma cases algorithm: Asthma diagnosis on ≥1 visit AND asthma diagnosis or medication presciption on ≥1 other visit AND no other chronic lung disease diagnosis on ≥2 visits AND no reported smoking history ≥10 years Initial asthma controls algorithm: No diagnosis for any respiratory disease or cancer AND no prescription of any astha/COPD/iimmunodepressant medication AND no reported smoking history ≥10 years Final asthma controls algorithm: ≥2 visits with any asthma diagnosis or prescriptions AND no diagnosis for any respiratory disease or listed cancer AND no prescription of any asthma/COPD/immunodepressant medication AND no reported smoking history ≥10 years	Manual review of 100 cases for both algorithms
Vollmer et al,^Citation26 2004 United States July 1998 to January 1999	KPNW, Epic, OSCAR, TOPS ED, secondary care	235,000 patients with continuous health plan eligibility aged 15–55 in January 1999 9,723 asthma patients identified	ICD-9 codes	Health care utilization profiles used for validation study 1. Four “high-probable” categories: → Two or more non-urgent care outpatient contacts for asthma → A single non-urgent contact and one or more ED or inpatient contact for asthma → Any Industrial Medicine visit for asthma → Any asthma visit and either of the two medication dispensing criteria 2. Single non-urgent outpatient visit only 3. Four or more β-agonists, with or without a nebulizer treatment order, but no asthma visits of any kind and no ICS dispensings 4. ED or urgent care visit for asthma and nebulizer treatment order, but no other medication criteria met and no other types of asthma visits 5. Hospitalization for asthma, but neither asthma medication criterion met and no outpatient asthma visits of any kind 6. ED or urgent care visit for asthma, but no other types of asthma visits and no asthma medication criteria met 7. Nebulizer treatment but no asthma visits of any kind and no other medication criteria met 8. All other cases	Criteria used in medical records review Probable asthma • Two or more asthma health care visits • A single visit for asthma with a chart notation indicating a prior history of asthma • A single health care visit for active symptoms of asthma (wheeze, cough, shortness of breath) • A single visit for an asthma exacerbation that responds to therapy, even if no prior history Possible asthma • Patient-reported history of asthma noted in chart, but no evidence of active asthma or treatment for asthma • An uncorroborated ED diagnosis of asthma • Diagnosis of “rule out asthma” with no clear resolution
Donahue et al,^Citation27 1997 United States	Harvard Pilgrim Health Care (HPHC); Primary, secondary and emergency care	Random sample of 100 patients	Asthma code	Asthma diagnosis and asthma drug dispensing	Manual review by clinicians
Premaratne et al,^Citation28 1997 United Kingdom 1994	Accident and EDs of two hospitals	All asthma patients January–March 1994 1,185 records, of which 209 did not have enough data	String containing “asth*”	String containing “asth*” in the free text records	Affirmation of asthma diagnosis: Final diagnosis of asthma by clinical officer OR symptoms of asthma and (history of asthma or bronchodilators given, with improvement) OR known asthmatic presented with symptoms or for medication Rejection of asthma diagnosis: Clear alternative diagnosis Sufficient other information to reject asthma diagnosis
Comparison to an in dependent database
Engeland et al,^Citation29 2009 Norway	MBRN: population-based birth registry, all births in Norway since 1967 (more than 2.3 million) NorPD: all dispensed prescriptions from January 2004 in Norway	108,489 pregnancies, of which 4,549 mothers were recorded as having asthma in MBRN	Asthma	Asthma diagnosis in MBRN	NorPD: asthma medication
Coulter et al,^Citation30 1989 United Kingdom	7 general practices in the Oxford community health project 2,199 patients on medication Primary care	2,443 on digital register Bronchodilators, inhaled CS, prophylactic drugs	Asthma diagnosis	Asthma diagnosis on register	Manual review against the list of patients on long-term medication
Comparison to a quiestionnaire
Ward et al,^Citation31 2004 United Kingdom 1995–2004	GP Practice with 14,830 patients 83 1 controls, 587 responses Primary care	833 asthma patients, 659 responses 16–55 years on 1 October 1997	Asthma in GP database	One of the following criteria: 1. Read coded “asthma” diagnosis, H33 2. Attendances recorded on the asthma care screen 3. An intervention for asthma recorded 4. A textual entry “asthma” or “wheez” in the medical history 5. Inhaled steroids in the repeat prescriptions 6. Inhaled bronchodilators in the repeat prescriptions 7. Cromolyns in the repeat prescriptions	Questionnaire to determine bronchial hyperreactivity Cases: asthma in database Asthma diagnosis and bronchial hyperreactivity: considered positive Asthma diagnosis without bronchial hyperreactivity: further investigated in GP record Controls: bronchial hyperreactivity but no asthma diagnosis

Abbreviations: CPP, cumulative patient profile; ICPI, integrated primary care information database; GP, general practitioner; EHR, electronic health record; SAGE, Study of Asthma, Genes and the Environment; KPNW, Kaiser Permanente Northwest Division; OSCAR, outside claims database; TOPS, The outpatient pharmacy system; ED, emergency department; ICS, inhaled corticosteroids; MBRN, Medical Birth Registry of Norway; CS, corticosteroids; NorPD, Norwegian Prescription Database.

Table 2 Characteristics of studies with validated asthma algorithms

Author, year, country, prevalence	Algorithm	Sensitivity, 95% CI	Specificity, 95% CI	PPV, 95% CI	NPV, 95% CI	Prevalence
Manual validation
Xi et al,^Citation19 2015 Canada	1. Asthma in disease registry	7% (5–10)	99% (97–100)	67% (38–87)	73% (72–74)	8.1%
	2. Billing code	77% (75–83)	89.2% (86–92)	74% (67–80)	91% (88–94)
	3. Asthma in CPP	63% (59–68)	92% (90–95)	76% (68–83)	87% (83–89)
	4. Asthma medications	79% (75–83)	64% (59–68)	46% (41–50)	88% (84–92)
	5. Asthma in chart notes	85% (81–88)	76% (72–80)	58% (52–63)	93% (89–95)
	6. Asthma in CPP OR billing code 493	90% (87–93)	84% (80–88)	69% (63–74)	96% (93–97)
	7. Asthma in CPP OR billing code 493 (exclusion codes 491, 492, and 496)	87% (83–90)	85% (82–89)	70% (63–76)	94% (91–96)
	8. (Asthma in chart notes OR asthma medications) AND billing code 493	78% (74–82)	92% (89–95)	79% (72–85)	91% (88–94)
	9. (Billing code 493 OR medications) AND asthma in chart note	84% (80–88)	84% (80–88)	67% (61–73)	93% (90–95)
	10. Billing diagnostic code 493 AND asthma in chart notes	74% (70–78)	93% (91–96)	81% (73–87)	90% (87–93)
Engelkes et al,^Citation20 2014 Netherlands	Definite, probable and doubtful cases			63%
Afzal et al,^Citation21 2013 Netherlands	Definite asthma	98%	95%	66%		6%
	Definite + probable	96%	90%	82%		29%
	Definite, probable and doubtful cases	95%	67%	57%		32%
Dexheimer et al,^Citation22 2013 United States	Algorithm constructed using a Bayesian network system			64%		7–10%
Wu et al,^Citation23 2013, 2014 United States	ICD-9 codes	31	93	57	82	4–17%
	Natural language processing: logic	81	95	84	94
	Natural language processing: machine learning	85	97	88	95
Kozyrskyj et al,^Citation24 2009 Canada	At least one asthma hospitalization, or two physician visits, or four prescription medications	47% (35–60)	92% (78–98)	91% (76–98)		11%
	At least one asthma hospitalization, or two physician visits, or two prescription medications	67% (54–78)	92% (78–98)	94% (82–99)
	At least one asthma hospitalization, or one physician visit, or two prescription medications	77% (65–87)	92% (78–98)	94% (85–99)
	At least one asthma hospitalization, or one physician visit, or two bronchodilators, or one controller medication	80% (69–89)	89% (74–97)	93% (83–98)
	At least one asthma hospitalization, or one physician visit, or two bronchodilators, or one bronchodilator and ketotifen or an oral steroid, or one controller medication	80% (69–89)	89% (74–97)	93% (83–98)
	At least one asthma hospitalization, or one physician visit, or one bronchodilator, or one controller medication	82% (70–90)	83% (67–94)	90% (79–96)
Pacheco et al, 2009 United States	Initial algorithm	70% (60–78)	100%	100% (90–100)	77% (65–86)	7.2%
	Final algorithm	95% (84–99)	96% (87–99)	95% (84–99)	96% (87–99)
Vollmer et al,^Citation26 2004 United States	Algorithm 1: population of 4460			95%		4.1%
	Algorithm 2: population of 2334			90%
	Algorithm 3: population of 545			70%
	Algorithm 4: population of 25			100%
	Algorithm 5: population of 11			50%
	Algorithm 6: population of 721			80%
	Algorithm 7: population of 99			27%
	Algorithm 8: population of 1528			80%
Donahue et al,^Citation27 1997 United States	Asthma code and drug dispensing			86%		3%
Premaratne et al,^Citation28 1997 United Kingdom	String containing asth* in free text records	80% (75–86)	96% (96–99)	91% (87–94)	94% (93–95)	20.6%
Comparison to an independent database
Engeland et al,^Citation29 2009 Norway	Asthma in MBRN and NorPD	51% (49–52)	98% (98–98)	46% (45–48)		4.20%
Coulter et al,^Citation30 1989 United Kingdom	Percentage of people on long term medication and recorded on the register			58%
Comparison to a questionnaire
Ward et al,^Citation31 2004 United Kingdom	Total of all reviewed patients			89%		5.60%
	Cases without bronchial hyperreactivity			73%
	Controls with bronchial hyperreactivity			78%

Abbreviations: PPV, positive predictive value; NPV, negative predictive value; NLP, natural language processing; ML, machine learning; MBRN, Medical Birth Registry of Norway; NorPD, Norwegian Prescription Database.

Table 3 Quality assessment using QUADAS-2

Study	Risk of bias
	Patient selection	Index test	Reference standard	Flow and timing
Xi et al,^Citation19 2015	☺	?	☺	?
Engelkes et al,^Citation20 2014	☺	☺	☹	☺
Afzal et al,^Citation21 2013	☹	☺	☺	☺
Dexheimer et al,^Citation22 2013	☺	☺	☺	☺
Wu et al,^Citation23 2013,2014	☹	☺	?	☺
Kozyrskyj et al,^Citation24 2009	☹	☹	☺	☺
Pacheco et al,^Citation25 2009	☹	☺	☺	☺
Vollmer et al,^Citation26 2004	☹	☺	☺	☺
Donahue et al,^Citation27 1997	☺	☹	☹	☺
Premaratne et al,^Citation28 1997	☺	☹	☺	☺
Engeland et al,^Citation29 2009	☹	☹	☹	☹
Coulter et al,^Citation30 1989	☹	☹	☹	?
Ward et al,^Citation31 2004	☹	☹	☺	☹

Note: Happy face: low risk; sad face: high risk; question mark: unclear risk.

MoherDLiberatiATetzlaffJAltmanDGPRISMA GroupPreferred reporting items for systematic Reviews and meta-analyses: the PRISMA statementBMJ2009339b253519622551

 PubMed Web of Science ®Google Scholar

XiNWallaceRAgarwalGChanDGershonAGuptaSIdentifying patients with asthma in primary care electronic medical record systems Chart analysis-based electronic algorithm validation studyCan Fam Physician20156110e474e48326759847

 PubMed Web of Science ®Google Scholar

EngelkesMAfzalZJanssensHAutomated identification of asthma patients within an electronical medical record database using machine learningEur Respir J2012404655

 Web of Science ®Google Scholar

AfzalZEngelkesMVerhammeKMJanssensHMSturkenboomMCKorsJASchuemieMJAutomatic generation of case-detection algorithms to identify children with asthma from large electronic health record databasesPharmacoepidemiol Drug Saf201322882683323592573

 PubMed Web of Science ®Google Scholar

DexheimerJWAbramoTJArnoldDHImplementation and evaluation of an integrated computerized asthma management system in a pediatric emergency department: a randomized clinical trialInt J Med Inform2014831180581325174321

 PubMed Web of Science ®Google Scholar

WuSTSohnSRavikumarKEWagholikarKJonnalagaddaSRLiuHJuhnYJAutomated chart review for asthma cohort identification using natural language processing: an exploratory studyAnn Allergy Asthma Immunol2013111536436924125142

 PubMed Web of Science ®Google Scholar

KozyrskyjALHayGlassKTSandfordAJParéPDChan-YeungMBeckerABA novel study design to investigate the early-life origins of asthma in children (SAGE study)Allergy20096481185119319416140

 PubMed Web of Science ®Google Scholar

PachecoJAAvilaPCThompsonJAA highly specific algorithm for identifying asthma cases and controls for genome-wide association studiesAMIA Annu Symp Proc2009200949750120351906

 PubMedGoogle Scholar

VollmerWMO’ConnorEAHeumannMFrazierEABreenVVillnaveJBuistASSearching multiple clinical information systems for longer time periods found more prevalent cases of asthmaJ Clin Epidemiol200457439239715135841

 PubMed Web of Science ®Google Scholar

DonahueJGWeissSTGoetschMALivingstonJMGreinederDKPlattRAssessment of asthma using automated and full-text medical recordsJ Asthma19973442732819250251

 PubMed Web of Science ®Google Scholar

PremaratneUNMarksGBAustinEJBurneyPGA reliable method to retrieve accident and emergency data stored on a free-text basisRespir Med199791261669122513

 PubMed Web of Science ®Google Scholar

EngelandABjørgeTDaltveitAKVollsetSEFuruKValidation of disease registration in pregnant women in the Medical Birth Registry of NorwayActa Obstet Gynecol Scand200988101083108919657758

 PubMed Web of Science ®Google Scholar

CoulterABrownSDanielsAComputer held chronic disease registers in general practice: a validation studyJ Epidemiol Community Health198943125282592887

 PubMed Web of Science ®Google Scholar

WardDGHalpinDMSeamarkDAHow accurate is diagnosis of asthma in a general practice database? A review of patients’ notes and questionnaire-reported symptomsBr J Gen Pract20045450775375815593441

 PubMed Web of Science ®Google Scholar