Initial Validity Evidence for the American Board of Emergency Medicine Emergency Medical Services Certification Examination

ABSTRACT

Objectives: The American Board of Emergency Medicine (ABEM) Emergency Medical Services Medicine (EMS) was approved by the American Board of Medical Specialties on September 23, 2010. Subspecialty certification in EMS was contingent on two key elements—completing Accreditation Council for Graduate Medical Education (ACGME)-accredited EMS training and passing the subspecialty certification examination developed by ABEM. The first EMS certification examination was offered in October 2013. Meaningful certification requires rigorous assessment. In this instance, the EMS certification examination sought to embrace the tenets of validity, reliability, and fairness. For the purposes of this report, the sources of validity evidence were anchored on the EMS core content, the examination development process, and the association between fellowship training and passing the certification examination.

Methods: We chose to use validity evidence that included: 1) content validity (based on the EMS core content); 2) response processes (test items require intended cognitive processes); 3) internal structure supported by the internal relationships among items; 4) relations to other variables, specifically the association between examination performance and ACGME-accredited fellowship training; and 5) the consequences of testing.

Results: There is strong content validity evidence for the EMS examination based on the core content and its detailed development process. The core content and supporting job-task analysis was also used to define the examination blueprint. Internal structure support was evidenced by Cronbach’s coefficient alpha, which ranged from 0.82 to 0.92. Physicians who completed ACGME-accredited EMS fellowship training were more likely to pass the EMS certification examination (chi square, p < 0.0001; Cramér’s, V = 0.24). Finally, there were two sources of consequential validity evidence—use of test results to determine certification and use of the resulting certificate.

Conclusions: There is substantial and varied validity evidence to support the use of the EMS certifying examination in making summative decisions to award certification in EMS. Of note, there was a statistically significant association between ACGME-accredited fellowship training and passing the examination.

Keywords:

• EMS
• certification
• validity

Disclaimer

As a service to authors and researchers we are providing this version of an accepted manuscript (AM). Copyediting, typesetting, and review of the resulting proofs will be undertaken on this manuscript before final publication of the Version of Record (VoR). During production and pre-press, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal relate to these versions also.

INTRODUCTION

Emergency Medical Services (EMS), including the medical oversight of prehospital care, has been inextricably linked to the specialty of emergency medicine since the inception of the clinical practice of emergency medicine in Alexandria, Virginia in 1961 (1). The American Board of Emergency Medicine (ABEM) was recognized by the American Board of Medical Specialties (ABMS) on September 21, 1979. More than 30 years later, the ABEM subspecialty of EMS Medicine was approved by the ABMS on September 23, 2010. The ABEM recognized that EMS involved a unique knowledge base and proposed EMS subspecialty certification in order “to standardize physician training and qualifications for EMS practice, to improve patient safety and enhance the quality of emergency medical care provided to patients in the prehospital environment, and to facilitate integration of prehospital patient treatment into the continuum of patient care” (2).

Subspecialty certification in EMS was contingent on two key elements—completing Accreditation Council for Graduate Medical Education (ACGME)-accredited EMS training and passing the subspecialty certification examination developed by ABEM. The first EMS fellowships were accredited by the ACGME in February 2013 and the first EMS examination was offered in October 2013.

The reasons to pursue EMS subspecialty status in the ABMS community were manifold. It was hypothesized that the ACGME accreditation process would standardize EMS fellowship education and elevate the standards for accredited programs. Having a nationally based standard for certification would create greater confidence in physicians claiming to have expertise in the discipline. Finally, there was the belief that applying greater rigor toward the standards for the field would elevate the sophistication of EMS research.

Meaningful certification also requires rigorous assessment (3). In this instance, the EMS certification examination sought to embrace the tenets of validity, reliability, and fairness. There are many potential sources of validity for the EMS examination that, for the purposes of this report, are anchored in the EMS core content, the examination development process, and the association between fellowship training and passing the certification examination.

METHODS

Although Messick’s (4) and Kane’s (5) models are sufficient for assessing the validity evidence for the EMS examination, we chose to use a more contemporary model of validity, as described by Bandalos. The approach used by Bandalos expands the sources of validity evidence beyond the traditional dimensions of content, construct, and criterion. [6] Following Bandalos, the sources for validity evidence included: 1) content validity (based on the EMS core content); 2) response processes (test items require intended cognitive processes); 3) internal structure supported by the internal relationships among items; 4) relations to other variables, specifically the association between examination performance and ACGME-accredited fellowship training; and 5) the consequences of testing. Using these five dimensions, the authors were able to describe the validity argument for the EMS examination in greater detail.

Content validity is based on the core content that was developed using information gathered from the EMS community. The first core content upon which the examination was based was developed under the auspices of the ABEM EMS Examination Task Force (2). That core content was subsequently used to create a model curriculum for EMS fellowship training (7). The core content was last updated in 2019 (8). The subspecialty certification examination content is influenced by allocations of items (multiple-choice questions) that emphasize certain areas of the core content. This distribution of test items is called the “examination blueprint” (9).

Validity evidence for response processes can be derived from the item development methodology. A valid item would require that certain cognitive tasks are used to correctly address a tested idea.

Sources of internal structure validity include the processes used to create and refine items, as well as the psychometric performance of items. Internal structure validity evidence is supported by the reliability of the examination items, as measured by point-biserial correlations. At the test level, internal structure validity is supported by Cronbach’s coefficient alpha.

Validity support is demonstrated when there are associations with other variables. In this case, ABEM was interested in performance on the examination as measured by the examination’s pass or fail results and its association with ACGME-accredited EMS fellowship training. Specifically, a positive association between physician performance and ACGME-accredited EMS fellowship was posited as providing validity support for the EMS certification examination. In addition, for a criterion-referenced examination, the basis for establishing a criterion can also contribute validity evidence. Examination reliability is often considered separately from validity. However, reliability is recognized as indirectly contributing to the validity of the internal structure of an examination. The consequences of testing (e.g., making a determination of certification) and use of the resulting credential provides evidence of consequential validity.

The EMS Certification Examination is a criterion-referenced test with 300 single-best-answer, multiple-choice questions (items). A test using criterion referencing means that if a standard or passing score is met, the physician will pass. There is no “curve” resulting in a pass rate that could, in theory, range from 0% to 100%. The examination is a secure examination delivered at professional testing centers every other year.

Emergency Medical Services fellowship training was strictly defined as completion of an ACGME-accredited training program. Non–ACGME-accredited EMS fellowships existed prior to ABMS recognition of EMS as a subspecialty in 2010. Graduates from these programs were eligible through a modified training and practice pathway but were not considered fellowship-trained for the purpose of this study. The reason for this classification was that graduates would not have trained with the fellowship core content developed by ABEM. This core content was ultimately incorporated into the ACGME-accredited fellowship training guidelines upon which the EMS certification examination was developed.

Analysis

The analysis of the five factors that provide points of emphasis for this validity assessment is largely descriptive and qualitative. For the quantitative analysis of the association between examination performance (pass or fail) and eligibility type (fellowship training or practice eligibility) a 2-by-2 chi square test was used. We used a pre-hoc alpha of p < 0.01 based on a sample size greater than 1,000 to determine statistical significance. Only first-time test-takers were included in this study. First-time test-takers are used by ABEM for most high-stakes examinations to serve as a reference group for its psychometric measurements (e.g., examination equating). Repeat test-takers tend to be a smaller cohort, which is a statistically unstable and less predictable cohort.

RESULTS

Content Validity

There is strong content validity evidence for the EMS examination. In February 2011, ABEM formed the EMS Examination Task Force, which later became the EMS Examination Committee, and eventually became the EMS Subboard. This progression was part of the normal maturation of specialties within the ABEM structure. One of the earliest assignments for this group was to develop the core content for EMS. Earlier work by the National Association of EMS Physicians (NAEMSP) and the American College of Emergency Medicine (ACEP) EMS Committee produced a core content document in 2010 that ABEM used in its application to the ABMS for EMS to become a subspecialty (10). The work by the NAEMSP and ACEP was developed by content experts and served as a foundational document for the EMS Examination Task Force. The ABEM core content, developed by the EMS Examination Task Force, was reviewed by clinically active EMS physicians through a job-task analysis survey which rated each topic's importance and frequency to the practice of EMS Medicine. This type of survey process is called a “practice analysis” or “job analysis.” After this practice analysis, the results were reviewed again by a panel of subject matter experts. The EMS core content was made public in December 2011 and was published in July 2012 (2). Undergoing core content development using this process provided substantial evidence for content validity. ABEM ensured that the core content was available to test-takers for two years prior to the administration of the first certifying examination.

The EMS core content then served several roles. It defined the clinical practice of the specialty of EMS; guided curricular material for fellowships; and defined the content that could appear on the certifying examination. The core content and supporting job-task analysis was then used to define the examination blueprint. An examination blueprint reflects the allocations of various content areas on the examination. The current examination blueprint for the EMS certifying examination is: Clinical Aspects of EMS Medicine (40%); Medical Oversight of EMS (30%); Quality Management and Research (15%); and Special Operations (15%) (9). All examination items were referable to the EMS core content and supported by material in either a peer-reviewed journal, an authoritative text, a statute, or published regulatory materials.

On March 13, 2019, the EMS Examination Committee approved modifications to the EMS core content (8). These revisions were similarly supported quantitatively by the results of a second job-task analysis survey that had been conducted in 2018. The 2021 and 2023 EMS certifying examinations were designed using the 2019 revised core content.

One of the purposes of fellowship training is to prepare physicians to become certified (4). Soon after the development of the original ABEM-developed EMS core content, a model EMS fellowship curriculum was developed based on the core content (7). Having the fellowship curriculum for ACGME-accredited programs share the same core content as the examination was another reason this study included ACGME-accredited programs only. The curriculum included all major headings and subheadings in the EMS core content.

Response Processes

A validity argument for response processes is when “test items tap into the intended cognitive processes” (6). The EMS examination item development process provides some validity evidence. All item writers for the EMS examination are subject matter experts with extensive experience and ongoing clinical activity. Item writers undergo ongoing training in the psychometric principles of item development. Items are initially designed from a testing point, which informs the “intended cognitive processes.” The testing point is integrated, in most situations, into a question stem that includes a relevant clinical scenario. In addition, many of the questions require multiple steps and sophisticated thought processing that reflect higher levels of cognitive ability using Bloom’s taxonomy (11). These practices contribute additional validity evidence for the EMS examination.

Internal Structure

Validity support was demonstrated when relationships among test item performances were consistent with the relationships anticipated from theory. The EMS certification examination had a high degree of internal reliability among items, as measured by Cronbach’s alpha with coefficients that ranged from 0.82 to 0.92 (Table 1). Factors that contribute to Cronbach’s alpha include the number of items and the average inter-item correlations (12). There were 300 items on the examination, which contributed to the high alpha coefficients. The ABEM also measured the point biserial of each item, which is similar to the Pearson correlation coefficient if one of the variables is dichotomous. The ABEM typically scores items that have a point biserial that is both positive and of a sufficient magnitude. The magnitude of the Cronbach’s alpha values and item selection using the point biserial measurement provided additional validity evidence for the use of the EMS examination.

Relations to Other Variables.

The relationship of test scores to other variables, such as performance differences among groups, can also provide validity evidence. In the case of the EMS examination, we hypothesized that test performance would be higher among graduates of ACGME-accredited EMS fellowships. Since 2013, there have been 6 examination administrations involving 1,362 first-time test-takers (Table 2). In this group, there were 769 physicians (56.5%) who had not completed ACGME-accredited fellowship training, of whom 566 passed (73.6%). There were 593 physicians (43.5%) who had completed fellowship training, of whom 547 passed (92.2%; chi square, p < 0.0001; Cramér’s, V = 0.24). These findings demonstrate a statistically significant measure of association between ACGME-accredited training and passing the EMS examination, with a small effect size.

Further validity evidence is provided by the method for setting the passing standard in a criterion-referenced examination. The first EMS examination in 2013 underwent a standard-setting study using a modified Angoff method (13). This approach anchored the passing standard to a formal, defensible process to determine the passing score (“cut-score”) that was based on the anticipated performance of a test-taker who was just able to pass the examination (a minimally competent candidate). For the purposes of validity evidence, standard-setting creates a relationship between the passing score and the examination construct that is more defensible than choosing an arbitrary pass rate. Standard-setting using the Angoff method involves having a diverse group of physicians take the examination and determine the probability that a minimally competent EMS-certified physician would correctly answer each question. The results for all questions are compiled and averaged to determine the cut score (14).

Result Use

There are two forms of consequential validity evidence—use of test results and use of the resulting certificate. First, ABEM used the results of the test in a summative way to award board certification. Recall that EMS subspecialty certification by ABEM was only permitted by ABMS after a multiyear critical review involving a detailed application process. The substantial validity evidence and the psychometric rigor of the examination allowed ABEM to confidently award certification.

Another form of consequential validity is the use of EMS certification as a credential. Although there are no formal studies, anecdotally, academic institutions and the ACGME acknowledge EMS certification as a meaningful credential. Specifically, ACGME standards require that fellowship leadership and core faculty must be EMS-certified. EMS certification is also acknowledged by state and federal regulatory agencies, especially for leadership roles.

DISCUSSION

This is the first report assessing the validity evidence for the EMS certifying examination. Validity evidence supports the contention that the test assesses what it purports to measure. In this case, it is primarily the medical knowledge in the subspecialty of EMS Medicine. Validity is an incremental and iterative process that is tantamount to building evidence in a court of law. No single data point creates validity. Rather, validity is an argument that amasses evidence to create defensibility and confidence in the measurement. We believe that the various forms of validity evidence and the association between test performance and completing an ACGME-accredited fellowship make a strong argument in support of the validity of the EMS examination.

Traditionally, validity has often been described using Messick’s model of validity, which emphasized content, construct, and criterion relationships. The arguments in this report easily fit into Messick’s model. More contemporary models for validity, such as the one we used, provide a more detailed, cognitive, and analytic approach to building validity arguments.

There are several reasons why the validity evidence in support of the EMS certifying examination is so robust. First, the field was well-established prior to its recognition by the ABMS. There were multiple versions of EMS core contents prior to the first ABEM version in 2011. In addition, there was an early prototype curriculum for fellowship training (15). Moreover, non–ACGME-accredited fellowship training was common within the specialty. At the time of application to the ABMS for subspecialty status, there were 62 non-accredited fellowships (10).

The validity of the core content was further strengthened by the inclusion of an initial practice analysis (job-task analysis) that was conducted by Alpine Testing Solutions, with whom ABEM consulted in the early development phase of the EMS examination. To allow test-takers sufficient opportunity to become familiar with the examination content, ABEM posted the core content for two years prior to administration of the first test. Another similar job-task survey of EMS practitioners was conducted in 2018. Job tasks analyses are repeated as the activity and knowledge base within a field change. The survey results were used to revise the core content in 2019. Consistent with prior practice and the desire to give the EMS community preparation time, the 2021 examination was the first time that the test was based on the 2019 core content.

Standard setting is the exercise by which a passing score is determined. The ABEM performed a standard setting exercise immediately after administration of the first examination. The first standard-setting panel was composed of members of the Examination Committee. Thereafter, all standard-setting panels involved ABEM-certified EMS physicians who were not on the Examination Committee to offer an independent assessment of the test. As more fellowships became ACGME-accredited and as more test-takers had completed ACGME-accredited fellowships, ABEM revisited the passing standard and conducted another standard setting exercise in 2017. Participants in the 2017 standard-setting exercise were independent of the EMS Examination Committee. Although a revised core content was adopted in 2019, it was felt that the existing standard was sufficiently stable and relevant to continue to serve as the cut score for the examination.

Another factor that led to the substantial amount of validity evidence was that ABEM used processes and principles that were consistent with testing best practices. The EMS Examination Committee used the same approach to item writing that ABEM used for the Qualifying Examination in emergency medicine. That approach reflected the principles expressed in the Standards for Educational and Psychological Measurement (16). The ABEM’s processes have also undergone further scrutiny by the National Commission for Certifying Agencies (NCCA) with a favorable outcome (17). The NCCA was formed in 1977 to develop standards for certification programs in health care. ABEM is the only ABMS medical specialty to be accredited by the NCCA.

LIMITATIONS

This study has several limitations. First, this analysis did not review every type of validity evidence. For example, there was no analysis of differential item functioning. Differential item functioning examines the impact of item performance among different groups (e.g., women or men) to determine whether an item contains any bias. Unfortunately, differential item functioning requires larger test populations. Likewise, there were physicians from other specialties (e.g., pediatrics and anesthesiology) who took the examination for whom test performance was not separately analyzed. This is because the sample sizes for these groups were too small to obtain meaningful results. We strictly defined fellowship training as ACGME-accredited, even though non-accredited fellowships were abundant prior to 2013. Including all forms of fellowship training would likely have decreased the magnitude of the association between fellowship training and passing the examination but is beyond the scope of this research. Finally, there was an absence of quantitative data looking at consequential validity. A study examining job opportunities for EMS-certified physicians compared to non-certified physicians would have provided additional validity evidence.

CONCLUSIONS

There is substantial and varied validity evidence to support the use of the EMS certifying examination in making summative decisions to award certification in EMS Medicine. Of note, there was a statistically significant association between ACGME-accredited fellowship training and passing the examination. ABEM and the EMS Subboard will continue to study the validity and reliability of the EMS examination.

ACKNOWLEDGEMENTS: The authors wish to thank Ms. Yachana Bhakta, M.P.H. for her assistance with this manuscript.

DECLARATION OF INTEREST STATEMENT: Drs. Reisdorff, Joldersma, Kraus, and Barton are employees of the American Board of Emergency Medicine (ABEM). Dr. Knapp is a member of the ABEM Board of Directors. Drs. Kupas and Daya are members of the ABEM EMS Subboard. ABEM receives revenue for administering the EMS certifying examination. Dr. Clemency is a member of the Accreditation Council for Graduate Medical Education Emergency Medicine Review Committee.

DECLARATION OF GENERATIVE AI IN SCIENTIFIC WRITING: “The authors did not use a generative artificial intelligence (AI) tool or service to assist with preparation or editing of this work. The author(s) take full responsibility for the content of this publication.”

Funding Sources: None

Disclosure statement: Drs. Reisdorff, Joldersma, Kraus, and Barton are employees of the American Board of Emergency Medicine (ABEM). Dr. Knapp is a member of the ABEM Board of Directors. Drs. Kupas and Daya are members of the ABEM EMS Subboard. ABEM receives revenue for administering the EMS certifying examination. Dr. Clemency is a member of the Accreditation Council for Graduate Medical Education Emergency Medicine Review Committee.

Table 1. Internal Reliability Measured by Cronbach’s Alpha (EMS Examination 2013-2023)

Year	2013	2015	2017	2019	2021	2023
Cronbach’s Alpha	0.82	0.90	0.89	0.88	0.87	0.88

Table 1 footnotes: EMS = Emergency Medical Services Medicine.

Table 2. Association between Fellowship Training and Examination Performance 2013-2023

Training	Pass (Rate), n (%)	Fail (Rate), n (%)	Total, n
EMS Fellowship	547 (92.2)	46 (8.2)	593
No EMS Fellowship	566 (73.6)	203 (26.4)	769
Total	1,113 (81.7)	249 (18.3)	1,362

Table 2 footnotes: EMS = Emergency Medical Services Medicine.