A comprehensive process of content validation of curriculum consensus guidelines for a medical specialty

Abstract

In this article, we outline an innovative and comprehensive approach to the development by consensus of curriculum content guidelines for a medical specialty. We initially delineated the content domain by triangulation of sources, validated a curriculum blueprint by both quantitative and qualitative methodology, and finally reached consensus on content by Delphi methodology. Development of curricular objectives is an important step in curriculum development. Content definition or “blueprinting” refers to the systematic definition of content from a specified domain for the purpose of creating test items with validity evidence. Content definition can be achieved in a number of ways and we demonstrate how the concepts of content definition or validation can be transferred beyond assessment, to other steps in curriculum development and instructional design. Validity in Education refers to the multiple sources of evidence to support the use or interpretation of different aspects of a curriculum. In this approach, there are multiple sources of content-related validity evidence which, when accumulated, give credibility and strength to curriculum consensus guidelines.

Introduction

The purpose of this article is to describe an innovative and comprehensive process of validating curriculum content guidelines for a medical specialty, using as an example, the process and results of a curriculum content validation process done with members of the Canadian Obstetric Medicine Curriculum Consensus Group (CanCOM). This process of curriculum content validation is applicable to any domain of Medicine. First, we will describe, as context, the specialty of Obstetric Medicine. Then, we present a brief review of the literature on curriculum development, content definition and validity evidence, before presenting our approach to content validation. The approach used consisted of three steps: (1) initial delineation of curriculum content, by document analysis of a textbook and two curricula, review of cases prospectively collected in the services of a major tertiary care center and review by five local subject-matter experts (SMEs); (2) validation of curriculum content by conducting a survey with all Obstetric Medicine content experts in Canada; and (3) obtaining consensus on the content guidelines using Delphi methodology. The importance of a comprehensive approach to curriculum development in any medical specialty is that it results in creation of a robust set of guidelines obtained by consensus and based on several sources of validity evidence. Such a blueprint may then serve as a powerful cornerstone for subsequent steps of curriculum development.

Table

Practice points

This article proposes an innovative and comprehensive literature-based three-step process to develop curriculum consensus guidelines and maximize content-related validity evidence. Content definition of the curriculum guidelines should be attained through triangulation from multiple sources. Mixed methods applied to the survey of SMEs yields highly informative feedback. Delphi methodology is a practical and effective approach to obtain consensus on major modifications to finalize the curriculum guidelines

Context

Curriculum development was of particular importance in Obstetric Medicine given its relative youth. This discipline focuses on the care of women with medical conditions in pregnancy. In developed countries, causes of maternal mortality have shifted from surgical to medical causes, given the increasing age of women at the time of their first pregnancy and the increasing complexity of medical problems seen in pregnancy. A specialist in Obstetric Medicine bridges the gap between the specialist in Obstetrics, who may be unfamiliar with some medical conditions preceding or acquired during pregnancy, and other medical specialists, who may be uncomfortable in addressing the physiological and pharmacokinetic changes which occur during pregnancy (Nelson-Piercy 2002; Walters 2004; Hadden 2006). Although medical conditions in pregnancy have been recognized as an important component of the body of knowledge that a specialist in General Internal Medicine (GIM) in Canada is expected to master, the great majority of GIM residents1 are not systematically instructed about medical problems in pregnant women, thereby failing to link the curriculum to certification requirements and health care needs, as advocated by experts in curriculum design (Harden et al. 1999; Harden 2002). These concerns were the major impetus for the development of a comprehensive approach to validation of curriculum consensus guidelines for Obstetric Medicine.

Curriculum development

According to Kern (2009), curriculum development should encompass everything from problem identification (Why is this curriculum necessary?) to the regular, planned evaluation of the effectiveness of the curriculum. The process of curriculum development should be conducted in six inter-related steps: problem identification and general needs assessment which includes addressing the needs of the patient population (patient-centered education); needs assessment of targeted learners; formulation of goals and objectives; selection of instructional strategies; implementation; and evaluation and feedback. With regard to Obstetric Medicine, the problem and the needs were clear: medical problems in pregnancy are an important cause of mortality and morbidity, society and official bodies have asked for better training and provision of care, and learners do not feel well equipped to meet these challenges (Lee 2003; Walters 2004; Card et al. 2006). Once one clarifies the nature of the problem and measures the magnitude of the needs, the focus becomes the formulation of specific goals and objectives to be followed by review or selection of instructional strategies. Kern puts an emphasis on the interrelation between all steps. Indeed, an ongoing effort to ensure that the curriculum matches both patients and learners’ needs is paramount to the ultimate success of the curriculum.

Content definition

Defining curricular content helps to ensure that each step in the instruction and assessment process adequately represents the subject matter (Newble 1992; Tombleson et al. 2000). This process, known as content definition or “blueprinting”, as described by Nunally in 1978, refers to the systematic definition of content from a specified domain, for the purpose of creating test items with a high level of validity evidence. Much of the recent literature on content definition concentrates on developing content-valid tests such as MCQ- or OSCE-based assessments (Bordage et al. 1995; Tombleson et al. 2000; Newble 2004; McLaughlin et al. 2005a). However, we argue that the concepts and methods of content definition can be applied to content definition in general, whether it is for other steps in curriculum development or for instructional design. In this article, the term blueprint will be used to refer to the product of a systematic process of defining the content of a domain or curriculum.

The size and scope of any given blueprint varies, depending on its aim and the size of the domain. Examples of larger blueprints include the Dutch Blueprint 1994 which aims to cover the entire undergraduate medical curriculum in the Netherlands. Adaptations had to be made for each discipline in order to effectively focus on essential objectives (Raghoebar-Krieger et al. 1999). Yet, even a small domain may require delineation of specific or specialized content. For example, when designing an instrument to measure caregiver burden among families of stroke survivors, the authors systematically identified the content, with input from experts on the appropriateness of each item to the domain (Grant and Davis 1997). The results of content definition can serve many purposes including: selecting appropriate curricular objectives and instructional tools, assisting with learning, improving assessment and providing content-related validity evidence.

Validity evidence

The importance of validity evidence is a theme that permeates every aspect of content definition. In assessment, validity evidence aims to provide support to the inference that the examination accurately reflects the true competence or knowledge of the student (Bordage et al. 1995). The current standards propose five sources of construct validity evidence including evidence related to content, response process, criterion, relationship to other variables and consequences (AERA 1999). According to many measurement experts, content-related validity is a fundamental source of validity evidence when seeking to assess academic proficiency (Bridge 2003, p 416; Downing 2003). With regard to the creation of a national curriculum in Obstetric Medicine, the first task was to outline in clear terms and within a logical structure what was previously an unchartered discipline of medical complications of pregnancy. Therefore, at every stage of content definition, validity evidence was sought and documented (Munro 2000). After implementation of the curriculum, validity evidence may be sought from other sources including correlation with other variables (for instance comparison with other Obstetric Medicine curricula) and consequences (student feedback, patient satisfaction).

Proposed approach to content validation

When reviewing the literature on development of curriculum guidelines, we did not identify any systematic approach to curriculum content validation. Based on the concepts briefly presented above, we propose the following three-step approach to validating content for curriculum consensus guidelines.

• Step 1: Initial delineation of curriculum content

Content can be defined in several ways: directly by SMEs; from published curricula or examination questions; based on an observational study that analyzes what the learner needs to master; or a combination of approaches (Tilden et al. 1990; Audiss & Roth 1999; Tombleson et al. 2000; Crossley et al. 2002; Vogt et al. 2004; McLaughlin et al. 2005a).

By reviewing the literature, and by contacting members of the North American Society of Obstetric Medicine, we were able to identify one published curriculum, two that were in the process of being developed and several recent textbooks. Clinical data was obtained on cases prospectively collected in the outpatient and inpatient services of a major tertiary care Mother-Child Health Centre over a period of six months. This source provided a framework on which was added content from (1) the topics covered by the Brown University (Powrie et al. 2000), (2) the ISOM curricula, and (3) a document analysis of Medical Care of the Pregnant Patient, edited by Lee in 2001 and in 2008. Based on triangulation from all these sources, we developed an initial blueprint of the important clinical characteristics, listed by discipline, relating to the care of pregnant women with medical conditions acquired during or prior to pregnancy.

A local review was performed with two sets of qualified SMEs, who had both clinical and pedagogical expertise in this domain, to create an instrument outlining potential content for Obstetrics Medicine that would subsequently be validated by a variety of additional methods. The validity of the result depends on the credibility and expertise of those reviewing the outline of content. The Standards for Educational and Psychological Testing puts an emphasis on the qualifications of the SMEs with attention given to training, experience, and publications (AERA 2001). There is no firm consensus on the number of SMEs required. Although obtaining input, with full consensus from a large number of SMEs and education experts may be ideal, in practice, the content validation approach depends largely on the purposes and external constraints such as availability of expertise, cost, and time. Whereas Lynn (1986) mentions a minimum of three, other authorities have recommended from 2 to 20 panel members (Tilden et al 1990; Grant & Davis 1997). Clearly, the actual number depends on the purpose of the content defining process, the stakes involved, and the necessity to include different perspectives. We proceeded in two steps, allowing for creation of serial drafts based on the feedback of SMEs. The first step included three individuals who practiced in the Health Centre where the blueprint was developed and who were responsible for the development, over 20 years, of an Obstetric Medicine rotation. The second step included review by two individuals who practiced in a different Health Centre in Quebec and who are also involved as teachers in this domain.

Orientation of SMEs to the purpose of the blueprint is fundamental, to engage participation and to achieve the desired result (Grant & Davis 1997). The SMEs were first oriented to the purpose of the curriculum, the nature of the information sought, a timeframe, and the reason for their being selected, all to clarify the role they are expected to play. SMEs were asked to assess each item in the blueprint for appropriateness and accuracy of content, for redundancy with other items, for being essential versus superfluous, and for style (whether the wording is clear and devoid of colloquialisms) (Berk 1990). SMEs were asked to suggest additions or deletions and provide an overall evaluation of the blueprint, with attention paid to comprehensiveness (whether the content domain adequately covers all aspects of the construct) and clarity (Grant & Davis 1997).

The content of our blueprint was expressed in short phrases that summarize either an objective or specific achievement domain. Wording and level of detail reflected the purpose and the size of the curriculum and was intended to be meaningful even to the outside reviewer. For example, we listed content by objectives required for managing each condition under each discipline (Figure 1). The use of hierarchical numbering facilitated review and access to the curriculum framework and indicated the relative importance of the content. The educational objectives were formulated in the context of the framework of core competencies disseminated by the Royal College of Physicians and Surgeons of Canada as the CanMEDS roles, which includes the roles of Medical Expert (the central role), Communicator, Collaborator, Health Advocate, Manager, Scholar, and Professional (RCPSC, 2005). Alternatively, a clinical blueprint may list content as symptoms or clinical presentations (chest pain), illnesses (angina) or in terms of testable material (EKG findings of ischemia).

Figure 1. Excerpt from the CanCOM blueprint of competencies for Obstetric Medicine.

Only content judged to be essential by all members of both groups was included. The final result was a 28-page survey containing 402 separate items divided into 12 disciplines: Hypertension, Cardiology, Endocrinology, Respirology, Hematology, Gastroenterology, Neurology, Nephrology, Infectious Diseases, Immunologic Disorders, Dermatology, and Psychiatric Disorders

• Step 2: Validation of curriculum content using survey methodology

Further validation of the content by a larger consensus group was the second stage of this comprehensive approach to the development of valid curriculum consensus guidelines. Both quantitative and qualitative methodologies were used for this step.

A curriculum consensus group was formed by contacting all physicians in Canada who taught and/or practiced Obstetric Medicine. Of the 27 eligible members 25 (we excluded the curriculum administrator and those who had been involved in the local review) provided feedback on the survey. Respondents were asked to indicate for each item, whether it should be included, modified, or deleted. They were also asked to provide justifications for modifying or deleting content. Finally, they were invited to indicate additional content they thought should be included.

Reliability was estimated by calculating the level of interrater agreement for each item and each option: include, modify, or delete. A content validity index was derived by calculating the proportion of experts who rated an item as representative or highly representative of the content domain. Finally, a kappa coefficient can also be computed as a more robust measure of agreement since it aims to test whether agreement exceeds chance levels (Hunt 1986). Several adaptations have been made to Cohen's (1960) initial work on reliability assessment, including Fleiss's kappa applicable for any fixed number of raters (Fleiss 1971). Randolph (2005) has pursued this work and proposes a free-marginal kappa for any number of cases, categories, or raters. A free-marginal kappa was chosen instead of the fixed-marginal kappa because responders were not asked to assign a certain number of items to each category (Brennan & Prediger 1981). Our free-marginal kappa ranged from 76% to 92%. Different authors have proposed different cut-off values for interpreting the kappa. Landis and Koch (1977) have proposed a table with the following cut-offs: kappa <0: poor agreement; 0.00–0.20: slight agreement; 0.21–0.40: fair agreement; 0.41–0.60: moderate agreement; 0.61–0.80: substantial agreement; and 0.81–1.00: almost perfect agreement. A rule of thumb is that a kappa of 0.70 or above indicates adequate inter-rater agreement (Landis & Koch 1977). However, none of these scales have been validated and should be used with caution (Maclure & Willett 1987).

Qualitative analysis of the justifications was performed using analytic methods association with grounded theory approaches. The qualitative data were analyzed inductively by the curriculum administrator, with themes formulated based on constant comparison to previous coding (Glaser & Strauss 1967; Harris 2003). Revision of the themes was made, as necessary, during the process of coding (Strauss & Corbin 1997). To increase validity evidence, a second coder (PG) recoded the entire data set using the same table of codes. Adjustments were made to the coding schema, by consensus, following this second coding. An inter-rater agreement was calculated. Overall, qualitative analysis of the comments made in the validation process for the curriculum content was critical in revealing the nature of problematic items, adding constructive insights to items that were not problematic but could be improved and eliciting opinions about additional content that could be included. The results from the qualitative analysis led to the creation of a Delphi survey to obtain consensus on the more important or controversial changes that could be made in the curriculum blueprint.

• Step 3: Obtaining consensus on modifications using Delphi methodology

Delphi methodology was used to obtain consensus with regard to the more important or controversial modifications to the curriculum blueprint, rather than having the authors decide about changes unilaterally. We identified nine problematic items defined, based on two pre-established criteria, one based on quantitative analysis – items with a percent agreement less than or equal to 80%; and one based on our subjective impressions – any other item for which the reviewers wished to obtain consensus on modification. In addition, one major theme in the qualitative analysis was perception of an unsatisfactory integration of the CanMEDS roles other than medical expert. For example, an item placed in Cardiology, highlighting the importance of counseling about the risks/benefits of breastfeeding when taking medications, was intended to focus on the competencies of Communicator and Health Advocate. This item was viewed by a majority of respondents as equally applicable to any other discipline. A proposal was made by one of the respondents to create separate sections to clearly delineate competencies other than Medical Expertise, to meet the standards of Canada's Royal College of Physician and Surgeons. Finally, nine new sections and 23 new conditions were suggested by one or more respondent. We wished to obtain consensus on which content to include or not in the final validated outline of curriculum content.

Delphi methodology allows for subjective input from a larger group, in an efficient manner across several time zones, and with potentially less influence by dominant personalities than a face-to-face meeting (Linstone & Turoff 2002). The methodology was adopted to respect the basic tenets of this type of structured communication: (1) possibility for individual feedback; (2) presentation of the group's assessment and views; (3) opportunity to revise one's view; and (4) some degree of anonymity.

Delphi survey participants were given clear instructions. They were asked to rate on a Likert scale of 1–5 their level of agreement with the proposed modification with indicating 1, strongly disagree; 2, disagree; 3, neutral; 4, agree; and 5, strongly agree. A space was provided with each item for comments to justify the rating (Figure 2). The curriculum administrator served as monitor and was responsible for compilation of the data and the distribution of each survey for each round.

Figure 2. Excerpt from the Delphi survey for consensus on additional sections.

After all the surveys were received, the mean, mode, median, and standard deviation were computed for each item. The comments were regrouped, without identification of the participant, and displayed in identical font and format. Certain items were said to reach consensus after Round 1 if the standard deviation was small, the comments similar in their recommendations, and further feedback was deemed unhelpful by the monitor. Each participant was then sent a second survey in which the mean, standard deviation, bar histogram, and list of anonymous comments were included for each item. The survey was customized for the remaining items requiring further consensus, by including the participant's previous ranking and a space for a revised ranking and comments. In the interest of transparency, the items that obtained consensus were left in the survey with all comments, the means, standard deviations, and bar histograms, along with the label, “consensus achieved”.

Computation of mean, mode, median, and standard deviation was undertaken in a similar fashion after Round 2. Again, certain items were removed from further inclusion in the survey if deemed by the monitor to have obtained consensus. Based on ongoing feedback and suggestions, revisions were made to the initial proposed modifications. The number of rounds was fixed ahead of time rather than determined by the number required to achieve a pre-determined level of consensus. This decision was based on two concerns: to avoid discouraging participants from engaging in a lengthy process and to avoid having participants reach an artificial consensus in an effort to just get the job done.

Discussion

This article describes a comprehensive process of content definition and validation, as one of the steps of curriculum development outlined by Kern (2009). For a relatively new content domain such as Obstetric Medicine, formulation of goals and objectives may present an important challenge and the first step in successful curriculum development. The process of content definition and subsequent validation can be useful in other content domains.

Validity evidence is derived from every step of the process: from content delineation, content validation, and consensus on major changes. First, the instrument which serves as basis for external validation can be created from several sources of reference: actual clinical data (job analysis), published or unpublished curricula, and document analysis of available textbooks. Second, the content should be validated by a survey of external SMEs. Finally, consensus on major alterations, using Delphi methodology, is a further source of validity evidence. In selecting the sources of reference, it is important to give due consideration to both patients and learners’ needs. Much literature exists on the importance of patient-centered curricula including in Internal Medicine (Ratanawongsa et al. 2011). It therefore was important for us to address patients’ needs by including the actual problems for which patients were consulting. In some instances, patient advocates could be included in the group of SMEs. We chose to address patients’ needs indirectly by triangulating our sources with the list of problems for which medical attention was sought. In addition, attention was given to include roles other than medical expertise by using the CanMEDs framework to address issues of communication, health promotion, and professionalism. Patients’ perspectives could be included in the review of such sections. Furthermore, addressing learners’ needs directly would confer additional validity evidence. We did not include residents in our sample, because without an existing curriculum and clinical exposure they were themselves faced with lack of knowledge on Obstetric Medicine as a discipline. However, with the arrival of this curriculum and structured exposure, resident feedback will be incorporated in the evaluation of current and improved versions.

The validity of the curriculum content may be confirmed by a high level of inter-rater agreement. The items with a low percentage of agreement to include may require revisions that can subsequently be adopted by consensus.

Qualitative analysis is an extremely useful adjunct to confirm the high level of inter-rater agreement, and the basis for agreement and disagreement. Furthermore, qualitative analysis results in the collection of information beyond what could be gleaned by quantitative analysis alone. Indeed, the objective of data collection in grounded theory is to obtain as many relevant perspectives as possible to address the research objectives (Kennedy & Lindgard 2006). Not only will this purpose be achieved, but one will also collect constructive input that can lead to improvements in the definition of content. Finally, qualitative analysis can generate a list of additional items to include. And finally, this approach can serve as an independent source of insight to delineate the nature of problematic items and ultimately led to the creation of the Delphi survey.

The Delphi survey method can be an important source of validity evidence in Medical Education. When well used, it is a rapid and effective way of obtaining the opinions of many experts, despite geographic dispersion, and in numbers beyond what can be achieved in face-to-face settings. In addition, because the survey is anonymous, there is less risk that a strong personality biases the input from other colleagues or that a group from a particular center dominates the debate.

The Delphi methodology presents some shortcomings that are worth noting (Jones & Hunter 1995; Hasson et al. 2000). First, the quality of the results depends on both the initial quality of the instrument and the objectivity of the monitor in translating the data. Second, Delphi surveys have a natural tendency toward consensus, as respondents contrast their ratings with that of the group. In addition, personality types have an influence on how one responds to dissent. To address the presence or absence of such an influence on the results, respondents can be surveyed after completion of the Delphi survey and asked what motivated most their changes in ratings, if any. Third, artificial consensus can result from poor presentation of the data, insufficient importance accorded to dissenting opinions, or simply because of fatigue with the process. This risk can be minimized by presenting all the comments in a random order, by preserving anonymity and by limiting the process to three rounds. At each additional round, there is a concern that consensus might be achieved artificially because of time constraints and fatigue with the process.

This overall approach to the development of curriculum content for any medical specialty is designed to be comprehensive and based on several sources of validity evidence. The validated product will serve as an extremely important tool for subsequent steps of curriculum development, whether development of curricular objectives, selection of instructional tools or assessment strategies, or curriculum evaluation.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Notes

Notes

1. In Canada, residents who wish to specialize in GIM complete an extra one or two years – after the 3 years of core Internal Medicine training – of Fellowship training in GIM before they are licensed to practice as General Internists.