Defining best practices and validation for curriculum mapping

Abstract

The alignment of curriculum content with standards is a quality improvement measure used to identify gaps or overlaps in academic content. Curriculum mapping results, while commonly used, are rarely reported. Undergraduate faculty mapped 11 courses in a single timepoint using a web-based curriculum mapping tool. Quantitative results were calculated and compared to defined client needs categories. A validation analysis was performed by mapping one course in triplicate. Faculty feedback following the exercise was obtained by online survey. Three client needs categories were determined to be marginally out of range in the initial analysis of the quantitative mapping data. The curriculum map was substantiated by high correlation of curriculum measures in the validation process. A curriculum map should be planned with the aim as the central guide to inform the standards and measures selected for comparison. Measures and Standards chosen for the map as well as any actionable data generated should be informed by the aim. Validation can be performed by mapping subsets of the curriculum in triplicate. A team approach with high participation from the faculty should be considered when planning a full curriculum map.

Keywords:

• Curriculum mapping
• nclex-rn
• quality improvement
• assessment

REVIEWING EDITOR:

• Serafina Pastore, Senior Editor, Education, Psychology, Communication, Università degli Studi di Bari Aldo Moro, Italy

SUBJECTS:

• Higher Education
• Curriculum Studies
• Nursing

Introduction

Curriculum mapping aligns academic course content with a set of standards to create a framework for identifying existing content gaps. Sometimes referred to as curriculum crosswalk, pacing guide or scope and sequence, the aligning of course content to standards is a frequent quality improvement measure for academic programs (DeBlieck et al., 2023; Harden, 2001). While curriculum mapping is common, we discuss here that the process approach is subject to methodological variability. Rarely described are detailed methods, quantitative or qualitative results or best practices for conducting curriculum mapping. Recent publications have highlighted their mapping of curriculum from specific programs to align with accreditation standards or knowledge, skills and abilities indices, we find that gaps in the process approach exist and new suggestions for methodologies and best practices will continue to improve these instrumental practices (Gulbis et al., 2021; Nance & Brown, 2022; Reekie et al., 2023). The purpose of this study is to describe our approach, report results and demonstrate one validation method. This study is intended to further expand the scope of curriculum mapping to begin to define a data-informed approach.

Curriculum mapping originated from the foundations of essentialism in the early 1900’s, when William C. Bagley called for standards and accountability in the American educational system (Null & Bohan, 2023). However, the term ‘curriculum mapping’ was not coined until the early 1990’s (Jacobs, 1991). To date, curriculum mapping has become an integrated and necessary tool in curriculum development and assessment. Electronic affordances to align curriculum to standards are becoming increasingly prevalent (HelioCampus, 2023; Exxat 2023) as they improve workflows and generate deeper and automated quantitative analysis. Traditionally, however, the curriculum mapping process has relied almost exclusively on the use of electronic spreadsheets, which requires manual analysis for curriculum gaps and overlaps in addition to significant data management for a team approach.

The mapping standards used in this report are derived from the published test plan from the National Council Licensure Examination for Registered Nurses (NCLEX-RN). Success on the NCLEX-RN exam is required for all graduates of nursing programs to become licensed RNs in the United States and Canada. An updated Next Generation NCLEX-RN (NGN), launched in April 2023. prompting this self-study of the Jefferson College of Nursing Bachelor of Nursing Science (BSN) curriculum and new exam standards (NCSBN, 2023). However, the advancement of many fields towards competency-based learning outcomes is also likely generating self-studies of academic programs to ensure appropriate curriculum coverage (Lammerding-Koeppel et al., 2018; Rawle et al., 2017).

Methods

This pilot study is a cross sectional quality improvement initiative with quantitative and qualitative components performed in 2023. A convenience sample of subjects consisting of 29 undergraduate nursing faculty was drawn from a large urban university- based college of nursing and academic medical center to perform curriculum mapping.

The curriculum content assessed in this study was derived from 11 courses and mapped to NCLEX-RN standards. The classroom-delivered content was categorized as Topics while simulation and laboratory activities were categorized as Applications, all of these are referred to as ‘Measures’ throughout this report. Appendix A from the NCLEX-RN Test Plan were selected as the Standards that curriculum Measures were mapped against (NCSBN, 2023). Academic Chairs and the Assistant Dean created the lists of topics and applications (Measures) using syllabi and topical outlines posted on the course webpages.

Procedures: full curriculum mapping

Faculty were invited to participate in one full day ‘curriculum mapping retreat’ where 11 courses were mapped. Morning sessions had six groups of faculty each paired with one dedicated mapper. In the afternoon, five groups of faculty mapped five courses, each paired with one dedicated trained mapper. The Exxat Data Import Team (see metrics) trained the mappers to use the platform. The mappers entered the mapping data as the faculty groups discussed the association of measures and with the standards. Mapping breakout groups had 90 minutes to complete the task. Five to six faculty were assigned to each group, two of those faculty were always past or current course leads having both taught as an instructor and been responsible for all aspects of course delivery, assessment, and experiential components. One week before the retreat, faculty received the list of standards derived from NCLEX-RN 2023 Test Plan that they would be mapping course measures against. The faculty participating in the retreat (29 individuals) were all full-time instructors in the 11 courses mapped and ranged from 0.5–24 years of employment in the college, with an average of 6.5 years and a median of 5.7 years. Curriculum mapping data compared to NCLEX Client Needs Categories is reported in Table 1.

Table 1. Summary results of curriculum measures identified by faculty at retreat.

Client needs categories (No. of measures identified across curriculum)	% Measures identified by faculty	Recommended NCLEX-RN percent (%)
Management of care, MOC (n = 1247)	16	15–21
Safety and infection control, SIC (n = 497)	6	10–16
Health promotion and maintenance, HPM (n = 1111)	14*	6–12
Psychosocial integrity, PSYI (n = 1437)	18*	6–12
Basic care and comfort, BSS (n = 516)	7*	6–12
Pharmacological and parenteral therapies, PPT (n = 1293)	16	13–19
Reduction of risk potential, RRP (n = 987)	12	9–15
Physiological adaptation, PA (n = 912)	12	11–17

*Indicates out-of-rage result.

Procedures: validation

To evaluate the Test-Retest reliability of the mapping process within the client needs categories, mappings were performed first by participating faculty at the curriculum retreat, then by two completely independent faculty members on the same course. The percentage of items mapped within each of the eight client needs categories was computed. Test-retest reliability is a measure used in research and psychometrics to assess the consistency or stability of a measurement instrument over time. It specifically examines whether the same results are obtained when the same client needs categories, within a curriculum, are mapped on separate occasions (Hassan, 2024). Common methods for assessing test-retest reliability including the Cronbach’s alpha coefficient (Cronbach, 1951), the Pearson correlation coefficient, and the Intraclass correlation coefficient (ICC) were employed. The Pearson correlation coefficient method measures the linear relationship between two sets of scores obtained from two separate curriculum mappings of the same course. The Pearson correlation coefficient ranges from −1 to 1, with 0 indicating no correlation, and a perfect correlation is indicated by 1. The Cronbach’s alpha coefficient measures the internal consistency, or reliability, of a set of survey items. Specifically, it is used to help determine whether a collection of items consistently measures the same characteristic. Cronbach’s alpha quantifies the level of agreement on a standardized 0 to 1 scale. Higher values indicate higher agreement between items.

Alternatively, the Intraclass correlation coefficient (ICC) method is commonly used for measures with continuous scores, such as scales or percentage of items mapped in various client needs categories. It estimates the degree of agreement between two sets of mappings obtained from the same curriculum at different times. The ICC ranges from 0 to 1, where higher values indicate better test-retest reliability. An overlayed a plot of the percentage of items mapped by client needs category for the set of three evaluators is included to visualize how well they tracked together.

Metrics

The mapping data was collected using the Exxat Prism platform (Exxat 2023). The Exxat Prism curriculum mapping module was chosen as the mapping software due to the ease of use of the Prism platform and customer service support from the Exxat Data Import Team who provided data-entry support for all standards and measures. Faculty identified the binary data as belonging to one of two categories: presence or absence of the measure across the curriculum. Binary data was entered into the mapping system by trained mappers, and coded measures were exported as an Excel file.

Survey data collection

One month after the retreat faculty received a Microsoft forms survey with feedback requested about the quality and organization of the facilitator as well has two open ended questions asking for recommendations for future curriculum mapping and feedback directly from the mappers. 21 faculty and staff responded to the survey over the course of 10 days (Table 2). Qualitative data consists of commentary during the retreat and faculty survey feedback after the retreat had concluded. Faculty responses were collected from a survey that asked the question, ‘What recommendations would you make for future successful curriculum mapping?’ The survey was conducted online and ensured the anonymity of the respondents.

Table 2. Selected faculty qualitative feedback.

1. I would have met with course leads before and identified content that needed to map, not copy the entire content outline.

2. Curriculum mapping to accreditation standards (in which case course learning outcomes will be mapped to the essentials).

3. I would suggest longer time sessions for the clinical course – the activity generated lots of discussion and significant view points.

4. It would be helpful to collapse all the requirements – accreditation standards and essentials to one list. Map the current content to the requirements.

5. I would ask that faculty choose which course/courses they would most want to be involved in with the curricular mapping.

6. At least one faculty member who teaches in the course should review the plan ahead of time. Sometimes things are not obvious on syllabus or to a person who doesn’t teach in the course.

7. It seemed like the mapping was subjective, therefore open to the interpretation of the participants, and some people are new to teaching.

8. We mapped topics and exemplars to the NLCEX blueprint that didn’t match anything. We left content on the table to tie to the new essentials, leaving some of the blueprint guessed.

Results

Mapping data

602 standards were included from the NCLEX-RN Test Plan and 468 measures were mined from the 11 courses mapped. In total 8000 measures were mapped to the standards. The results of curriculum mapping of 11 courses of the Jefferson College of Nursing (JCN) BSN curriculum to NCLEX-RN provided a highly comparative view of congruency with testing standards. Table 1 shows the percentage range of course content measures compared to the published range of test questions within each NGN- defined category. In total 11 courses were mapped against 602 NCLEX-RN standards. JCN was within the range of five NGN categories and outside the range of three. This would suggest that the BSN curriculum could be over- and/or underemphasizing a portion of the content in the NCLEX-RN defined test plan. For the faculty of Jefferson College of Nursing it was important to view these data with some amount of inquiry as the mapping process itself yielded the potential for discrepancies in the map provided (Discussion). These discrepancies prompted the validation study, where a single course was remapped two additional times.

Validation data

Figure 1 and Table 3 show that the distribution of items mapped are similar among the three evaluations. The minimum values range from 1 to 3 and the maximum values range from 23 to 24. Table 3 provides visual evidence that the three separate evaluations yielded consistent results in terms of the percentage of items mapped within each client needs category. For instance, when a client needs category yielded a small percentage of items mapped for one evaluation, it tended to rank low for the other evaluations. There was also strong internal consistency (Cronbach’s alpha = 0.958) and a strong correlation between each pair of evaluators in terms of the percentage of points attributed to each of the eight client needs categories [Pearson Correlation coefficient: (rho = 0.841 between evaluator 1 and evaluator2; rho = 0.848 between the Retreat and evaluator1; and rho = 0.960 between the Retreat and evaluator2] (Table 3). The Intraclass Correlation for inter-rater reliability (ICC) among the percentage of points mapped into each client needs category among the three evaluators was 0.962 (p-value <0.001).

Figure 1. Overlay plot of the validation data as a percentage of items mapped by client needs category. Client needs categories as defined by the NCSBN are Management of Care (MOC), Safety and Infection Control (SIC), Health Promotion and Maintenance (HPM), Psychosocial Integrity (PSYI), Basic Care and Comfort (BCC), Pharmacological and Parenteral Therapies (PPT), Reduction of Risk Potential (RRP), and Physiological Adaptation (PA). Client needs categories are mapped across the x-axis with each mapping event represented by as symbol and on the y-axis as a percentage of the total items mapped within the category.

Table 3. Correlation matrix assessing the strength of the association in the validation study.

Pearson correlation coefficients
(p-Value)
	Retreat	Evaluator 1	Evaluator 2
Retreat	1.00	0.848	0.961
		(p = 0.0078)	(p = 0.0001)
Evaluator 1	–	1.00	0.841
			(p = 0.0089)
Evaluator 2	–	–	1.00

Study limitations

The plan to complete the map at the retreat in a single timepoint was limited by the allocated lead time. The mappers could have used more time to practice mapping on the electronic platform and better prepared for keeping a group of nursing faculty on topic during group sessions. The compressed timeline also resulted in a faculty that was not provided transpicuous objectives for their goals for the retreat. The lack of orientation may have influenced the data collected (see Discussion); an assertion further supported by the qualitative faculty feedback (Table 2). While the timeline did not allow the planners to coordinate a feedback mechanism from time-constrained faculty, the participation of the large group of experienced instructors was necessary and strengthened the data collected. Other limitations include a lack of best practices for curriculum mapping and the inexperience with the Prism Exxat curriculum mapping module.

Discussion

The results of curriculum mapping yielded data that could be directly compared to the distribution of content of the NCLEX-RN Test Plan client needs categories (NCSBN, 2023). The Jefferson College of Nursing BSN curriculum was shown to be appropriately within the range of five Client Needs categories, and outside the range of three (Table 1). These analyses can be interpreted to indicate that the areas that JCN needs to improve curriculum content to more accurately prepare students to take the 2023 NGN. However, it’s important to analyze results speculatively as curriculum mapping, while informative, has drawbacks.

The literature review conducted identified very few publications that described best practices, methodologies, or any detailed reports for curriculum mapping. Higher-education curriculum mapping reports may be less common in the literature, representing as little as 30% of all scholarly papers that mention curriculum maps (Rawle et al., 2017). Notably, Gulbis et al. reviewed their methodology for curriculum mapping, providing quantitative and qualitative analysis and call for the need for more rigor and reporting of curriculum mapping (Gulbis et al., 2021). Indeed, we presume that curriculum mapping for quality improvement and review measures are completed by most if not all accredited programs in addition to institutions that are concerned with their academic and student success and yet very few disseminate their methodologies or findings (Rawle et al., 2017).

Survey responses suggested that faculty didn’t understand why the mapping was focused on anything other than accreditation standards (Table 2). This is also discussed in the limitations of the study because faculty were not well informed of the goals. These realizations, as well as the review of the available literature sparked conversation about future curriculum mapping and best practices.

The purpose for a curriculum map should inform the standards and measures selected for alignment. If the purpose of the institution is to map to accreditation standards, then course and program learning outcomes should be the measure. Accreditation standards describe broad, overarching learning goals and program and course learning outcomes similarly reflect broad goals of the respective curriculum. Our observations suggest that these types of accreditation-driven curriculum studies are the most common types of curriculum mapping exercises that institutions participate in. Just as valid in program assessment, especially for professional programs that result in a credential, is the alignment of curriculum with licensing examination content. While we are aware that these maps may be commonplace, the practice is unfortunately not widely disseminated. Credentialing and other professional board exams typically publish the content areas that they evaluate; these often describe granular content specific to the knowledge, skills and ability that the learner should be able to demonstrate. The curriculum measures to be mapped to granular credentialing exam topics should also be as granular. In the example provided here, the curriculum measures were derived from course topic outlines. Purposes for curriculum mapping such as changes in student success outcomes, board examination performance, or planning for a prospective substantive change in the curriculum or examination content, should all inform the choice of standards and measures.

Curriculum mapping associations during the retreat described above were decided by faculty groups that had a professional stake in the courses being mapped. As the courses are analyzed in a seemingly quantitative way, it’s highly conceivable, and even anticipated, that faculty will unconsciously include mapping associations that they wouldn’t if they were mapping a curriculum that they didn’t feel some responsibility. Observations from one curriculum mapper in this study explained that faculty made comments such as ‘we should take credit for that’, ‘we do mention that’, and ‘I know we talk about it’ when discussing the mapping associations with their faculty mapping group. Faculty in some groups seemed to make as many associations between their course content and the NGN as possible. It is conceivable that during the mapping retreat faculty felt that measuring more outcomes would make their courses look more comprehensive. While these observations of personal faculty investment are likely well known in academic communities of many types, best practices, and suggestions to circumvent these biases, we have found, are not clearly defined. The assignment of multiple faculty members per mapping group at the retreat was intended to reduce bias, however it is imaginable that many of these mapping measures are exaggerated. An example of this possible bias in our retreat data is that one course had over 2000 NGN standards mapped. There were 602 NGN standards entered into the mapping system and in this one course’s data set, 123 of these were not mapped to any of the course measures while some were mapped up to 18 times. These overlaps in one course associated with a single NCLEX standard can be explained in more than one way: (1) the course is redundant and needs to be assessed for too much material focused on one topic area; (2) faculty overestimated their mapping measures; or (3) that the topic is essential in the course material and should be covered in the context of increasing levels of learning. If the goal of the study is to make meaningful curriculum changes, then these overlaps should be analyzed in more detail.

As mentioned above, the raw data resulting from the curriculum mapping retreat initially seemed erratic with hundreds and even thousands of points mapped in a single course. To validate the process, a single course was mapped again two additional times. Comparing the total points mapped between repeated instances of curriculum mapping of the same course, revealed a great deal of inconsistency because some individuals tend to be much more conservative when mapping points within client needs categories than others. However, when we normalize the data by computing percentage of points mapped into each category, we significantly reduce the individual evaluator bias and significantly improve the test-retest reliability of curriculum mapping. Furthermore, when a team of faculty are employed to map curriculum, they tend to keep each other honest.

The faculty feedback survey was informative (Table 2) for our continued process improvement. Many of the faculty indicated that they should have been involved in the extraction of the course topics and applications that were measured against the NCLEX. Others criticized the use of topics and applications favoring learning outcomes, suggesting that they didn’t understand the rationale for the choice of measures and standards. The faculty in this study should have been better informed of the project goals and the basis for the selection of the measures and standards.

It is the hope of these authors that academics will review results and processes also begin to publish their own curriculum mapping data. Because curriculum mapping is a common process for academic program evaluation a set of best practices should be developed. We hope that our report will contribute to this gap.

Acknowledgement

The authors would like to acknowledge Chandani Parikh, Exxat, Inc., Customer Success Specialist, who provided significant support in the entry of measures and standards and who always responded quickly and clearly to all emails and requests. The authors would also like to acknowledge Karen Waterfall, MSN and Amy Joachim, MSN who assisted with the validation study. Publication made possible in part by support from the Thomas Jefferson University Open Access Fund.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Hannah R. Smith

Hannah Smith is an Assistant Professor and Assistant Dean for Academic Strategy at the Jefferson College of Nursing in Philadelphia, Pa. She has a PhD in biochemistry and molecular biology and has over 13 years of experience in curriculum development and program assessment. Her interests include academic technology, competency-based learning, and curriculum development.

Jesse Chittams

Jesse Chittams received his master’s degree in Mathematical Statistics in 1993, and has over 30 years of experience directing the data management and statistical efforts for research projects involving medical and academic records data within the following institutions: Drexel, UP ENN, NIH, and Jefferson. He currently serves as Consulting Director of Data Analytics for the Jefferson University School of Nursing. His research interests include longitudinal data analysis and clinical trials.