ABSTRACT
Phenomenon: The integration of public health (PH) competency training into medical education, and further integration of PH and primary care, has been urged by the U.S. Institute of Medicine. However, PH competencies are numerous, and no consensus exists over which competencies are most important for adoption by current trainees. Our objective was to conduct a group concept mapping exercise with stakeholders identifying the most important and feasible PH skills to incorporate in medical and residency curricula. Approach: We utilized a group concept mapping technique via the Concept System Global Max (http://www.conceptsystems.com), where family medicine educators and PH professionals completed the phrase, “A key Public Health competency for physicians-in-training to learn is …” with 1–10 statements. The statement list was edited for duplication and other issues; stakeholders then sorted the statements and rated them for importance and feasibility of integration. Multidimensional scaling and cluster analysis were used to create a two-dimensional point map of domains of PH training, allowing visual comparison of groupings of related ideas and relative importance of these ideas. Findings: There were 116 nonduplicative statements (225 total) suggested by 120 participants. Three metacategories of competencies emerged: Clinic, Community & Culture, Health System Understanding, and Population Health Science & Data. Insights: We identified and organized a set of topics that serve as a foundation for the integration of family medicine and PH education. Incorporating these topics into medical education is viewed as important and feasible by family medicine educators and PH professions.
Phenomenon
There is a movement under way to strengthen public health (PH) training for physicians. In 2002, the Institute of Medicine (now the National Academy of Medicine) recommended that all medical students receive basic PH training in population-based prevention and that a significant proportion of medical school graduates be trained in PH at a master of public health level.Citation1 In 2007, the Institute of Medicine/National Academy of Medicine reaffirmed its recommendation and added that all medical students should also receive training in leadership, emergency preparedness, and clinical and community preventive services.Citation2 There is a broad understanding that contemporary health challenges facing physicians will not be solved by clinical care alone.Citation3 In our increasingly complex healthcare environment, physicians will need to integrate population-based management strategies and work with a broad range of health professionals to optimally meet direct care needs and improve patient health.
Responding to a 2003 initiative of the Association of American Medical Colleges (AAMC) and the Centers for Disease Control and Prevention to support the integration of PH into medical student and residency education, multiple schools instituted curricular changes resulting in an increased emphasis on PH competencies and care of populations.Citation4–9 In addition to curricular changes, organizational task forces created frameworks that emphasized the importance of interprofessional population health curricula in health professions training.Citation10–14
Despite these initiatives, it remains unclear how to optimally integrate PH teaching and which competencies to emphasize or include in medical education curricula. In the 2014 AAMC Medical School Graduation Questionnaire,Citation15 27% of students reported their PH instruction was “inadequate,” up from 22% the year before. A greater proportion of students reported inadequate coverage of some PH topics, such as environmental health, health policy, global health, health surveillance, and disaster management, compared to others topics, such as health disparities, women's health, or basic PH sciences like epidemiology and biostatistics.Citation16 In addition, a minority of U.S. medical students reported field experience in community health or participation in service learning, and the proportion of students with community health experience has dropped 5% since 2010. Although more than 69% of students participated in faculty-supervised research projects, only 29% have had community-based research experiences.
The purpose of this study was to conceptualize the essential PH competency content that family medicine (FM) educators and PH educators perceive as important for medical learners to acquire. Given that several barriers to PH education in a medical context were cited in a previous survey of FM educators, including a lack of time, funding, preceptor expertise, and student interest, prioritizing which competencies receive focus may be vital. To achieve this aim, we used Concept Mapping,Citation17,Citation18 a mixed-method, participatory research approach that combines qualitative and quantitative research methodologies to characterize, describe, and explain the essential PH competencies for medical learners. We also sought to determine which competency content was perceived to be the most important and feasible for integration into medical education curricula.
Approach
We engaged multiple stakeholder groups, including teachers of FM, medical students, administrators, and PH experts, to conceptualize the most important and feasible PH competencies to address in medical education. Group Concept Mapping generally consists of the following steps: (a) brainstorming, (b) idea synthesis, (c) sorting and rating of ideas, (d) multivariate analyses, and (e) interpretation.Citation17,Citation19–22 illustrates the process and participants at each step of the group concept mapping method and specifies and describes the groups that participated in each step. Because not all participants shared identifying information and all responses were anonymized, exact group overlap is impossible to ascertain. We used the Concept System Global MAX, a web-based application specifically designed to support the data collection, analysis, and representation of the group concept mapping process and results.Citation23 The project was granted an exemption from review by the Institutional Review Board of SUNY Upstate Medical University (FWA #00005967, IRB #00000391).
Figure 1. Process flow chart indicated steps and number of participants at each step. Note: GCM = group concept mapping.
Table 1. Participants in each phase of the project.
Data collection and analysis procedure
Brainstorming
Generation of the content was accomplished in two steps. First, attendees at dedicated sessions held during two conferences hosted by the Society of Teachers of Family Medicine participated in a brainstorming exercise facilitated by a subset of the core research team (the authors, abbreviated as CRT). An estimated 40 attendees (approximately 20 at each session) had the opportunity to respond to an open-ended focus prompt and provide up to 20 ideas (i.e., phrase or short sentence) that completed the following phrase: “A key public health competency for physicians in training to learn is ….”
This process generated 48 specific ideas that served as “seed content” for the next step. Second, additional participants were recruited from Society of Teachers of Family Medicine listservs and via snowball sampling techniques applied to the contact lists of the investigators on the research team. These participants were sent a link to the Concept System Global MAX website, where they were asked to provide responses to the focus prompt. Participants were also asked to provide anonymous demographic information, including their current professional role, training and experience in PH, highest academic degree(s), age, race, ethnicity, and gender. We intentionally oversampled FM educators for the brainstorming process in order to focus on the teaching of PH competencies specifically within the context of FM. We intentionally did not specify whether we were asking about medical students or residents, so as to attract a broad range of suggestions. All brainstorming was performed independently by each brainstormer, without guidance from any CRT member.
Idea synthesis
We established the final list of statements necessary for creating the concept map through the process of “idea synthesis,” which minimizes redundancy and maximizes clarity of statements in an oversaturated sample using standard principles of text analysis.Citation24 The idea synthesis process began at the close of the online brainstorming and included the statements generated at the conference sessions, as well as the online activity. The lead researcher examined the full set of ideas obtained for clarity, repetition, and redundancy of language or meaning. Multiple ideas contained in one entry were parsed into separate statements, and multiple statements describing a common thought were collapsed into a single statement that best represented the redundant statements. Subsequently, a spreadsheet of a list of 225 items was provided to all core research team members except the mapping methodologist (abbreviated as CRT(a)). Each reviewed the list and recommended the final set. The lead investigator compiled all responses from team members and redistributed. Two conference calls were held with team members to discuss and reach agreement on the final set of statements. Finally, each complete statement was edited for spelling and grammar.
Sorting and rating of ideas
One month after the online brainstorming closed, a subset of the brainstorming participants (n = 42) accessed the sorting function of the Global MAX website, where they were instructed to sort virtual cards that contained each of the 116 items. Participants were directed to arrange the cards according to their similarity, grouping them into piles “in a way that made sense to them.” The participants received the following guidelines for the sorting task: (a) all items could not be put into a single pile, (b) all items could not be put into their own separate piles (although some items could be grouped by themselves), (c) items could not be placed in two piles simultaneously, and (d) there could not be any “miscellaneous” piles.
For the rating activity, a subset of 39 brainstorming participants responded to the 116 items set in a questionnaire format and asked to rate the items using a 5-point Likert-type response scale for two ratings: importance (specifically, the relative importance for learners to acquire) and feasibility (specifically, the relative ease in integrating into the medical curricula).
The sorter and rater groups most likely overlapped significantly, although the exact level of overlap is impossible to determine due to anonymization techniques employed. All sorting and rating was performed independently by each sorter and rater, without guidance from any CRT member.
Data analysis
The concept mapping analysis was conducted using the Concept System Global MAX web-based application. The application converted each participant's sort data into a 116 × 116 item binary square “similarity matrix,” into which a value of 1 or 0 was entered into the cell to indicate statements were or were not grouped together. All individual “sort matrices” were then aggregated by the software to form a total similarity matrix, where the cells contained the number of participants who sorted the items together in a pile. Hence, values between 0 and 42 were possible. This matrix was analyzed using nonmetric multidimensional scaling (MDS) analysis with a two-dimensional solution.Citation25 The MDS analysis yielded a two-dimensional (x, y) configuration of the set of statements, based on the principle that statements piled together most often are located more proximally in two-dimensional space, whereas those piled together less frequently are farther apart. The MDS analysis of the total similarity matrix was assessed via convergence and final stress value. As an indicator of the goodness of fit between the two-dimensional configuration and the original similarity matrix, a lower stress value reflects a stronger relationship between the optimal and actual configurations.Citation26 MDS-produced X–Y coordinates were used as the input for the hierarchical cluster analysis utilizing Ward's algorithmCitation27 as the basis for defining a cluster.
There are no specific, definitive criteria by which the final number of clusters can be selected. The approach used here was to start with the most frequent number of piles sorted by the 42 participants (mode = 10) and generate all cluster solutions that were two higher and lower. Thus, for the purpose of narrowing the choices to consider, cluster solutions of eight through 12 clusters were evaluated separately by the lead investigator (CM) and the methodologist (SR) as to whether for each configuration the merger of clusters appeared to adequately represent the data as stated, organized and prioritized by the participants. Subsequently, the core research team, supplemented by three PH graduate students (one in a combined MD/MPH program) and one additional FM faculty member with preventive medicine teaching experience and expertise, met for an interpretative discussion, examined the aforementioned cluster solutions, and came to consensus on the final configuration. The additional students and faculty served as a validity check on the content and constructs represented in the final cluster solution. The relative merits of the different cluster solutions were examined in detail, with the group determining the final cluster solution and labeling of the map that preserved the most detail yet still yielded coherent clusters of statements. As a final step in the visual content analysis, the clusters were further grouped into three metacategories by the lead investigator and approved by the investigative team based on potential contextual application of the various skills and competencies.
In addition to the visual content analysis of the clusters, individual statements within each cluster that were ranked highest in both importance and feasibility were examined using a bivariate scatterplot contrasting the ratings of importance and feasibility. The scatterplot enabled the identification of high priority statements within each category—in other words, the PH skill areas that are both important and feasible to incorporate into medical education and training. A pattern-matching graphic was also generated to explore the correspondence of the group concept mapping structure. Graphically, the pattern match was portrayed using a “ladder graph” that consisted of two vertical axes, one specified for the ratings of importance and the other for feasibility. The vertical axes were joined by lines that indicate the average values for each cluster on the concept map. Statistically, the two patterns were compared with a Pearson product–moment correlation that is displayed at the bottom of the ladder graph.
As a means of ascertaining and crystallizing the consensus of a group of stakeholders, group concept mapping does not hold generalizability, external validity, or similar constructs as goals. The principle aim is to generate a set of concepts and a map that reflect the view of the group that has participated. In this sense, several validation steps occurred. As an external check on the views of FM educators, and particularly as a check against the participation of the core research team (CRT(a)) in the brainstorming, sorting, and rating steps, we included outside voices in the final interpretative process (three students and an available representative from the FM educator community). In addition, we invited all participants who had taken part in at least one phase of the project to participate in a review of the findings, through a reading of the draft of the report. Sixty-three individuals participated as members of the FM/PH Competencies Working Group. This represented a member-checking step to ensure that the final interpretation matched the intentions of the brainstormers, sorters, and raters. As a final validation step, it is common in group concept mapping to compare the results obtained by the stakeholders involved in the process with similar frameworks, studies, literature, or other sources of comparison. Following common practice, we compared our final cluster and metacategory results with the competencies described in six other similar reports.Citation12,Citation14,Citation28–31
Findings
Ninety-nine brainstormers generated 225 original ideas, which were then condensed, edited, amended, and modified to 116 meaningful and nonduplicative statements through the idea synthesis process. The MDS analysis of the total similarity matrix of the sort data from the 42 sorting participants converged after 30 iterations, producing a final stress value of 0.28, which is similar to that of previous meta-analyses examining typical concept mapping studies.Citation21,Citation32 The full set of all 116 final statements, along with individual ratings and ratings by cluster, are available via the Society of Teachers of Family Medicine Resource Library.Citation33 The background characteristics of the brainstorming and sorting/ratings participants are listed in .
Table 2. Main role, education, and public health background of panel participants.
As determined by MDS, the positioning of the points represents the way in which the group sorted the individual statements. On the map, statements viewed as more related to one another are closer together, whereas statements viewed as conceptually distinct from one another are located farther apart. Each shaded region represents a group of statements and is labeled to the conceptual content derived from the items contained within its boundaries. The final labeled 12-cluster solution, determined to be the most parsimonious description of the content by the research team and students, is presented in . The final cluster solution is similar to the median number of sorted piles from the 42 sorters (Mdn = 11.5) and is consistent with previous research suggesting a relationship between piles and final clusters.Citation21
Figure 2. Twelve-cluster solution, representing each cluster, statement numbers, and metacategories identified through the group concept mapping process.
The proximal-distal relationship of points, and by extension groups of points distinguished by cluster boundaries, is conceptually meaningful when interpreting the concept map. Each cluster derives meaning from the points within, and the location of clusters provide further information as to relative relationship with other clusters. As a function of how participants sorted the statements, points and clusters closer together are more conceptually similar than those farther apart. Utilizing this aspect of the group concept map, the interpretive discussion suggested the presence of three metacategories based on the arrangement of the 12 clusters ():
Clinic, Community & Culture—representing “Clinical Prevention,” along with “Cultural Competence,” “Social Determinants of Health,” and “Community Engagement” clusters. These relate to skills and knowledge that future clinicians would employ in direct patient interactions, in the management of their clinical populations, and in interactions with their immediate community.
Health System Understanding—including “Public Health Roles,” “Health Systems and Policy,” “Inter-sectoral Cooperation,” and “Public Health Foundations.” This metacategory represents knowledge areas that speak to an overarching understanding of PH and healthcare systems.
Population Health Science & Data—including “Research Application,” “Data Science,” “Surveillance,” and “Environmental Health.” This metacategory represents the application of population health methods and knowledge to clinical practice, including research methods, epidemiology, and statistics.
The rating data were averaged across the 39 participants for each item and cluster, and the mean ratings across participants, items, and clusters were fully integrated in the concept mapping output to examine different value patterns across the conceptual structure. Within each cluster and metacategory, a number of the original brainstormed statements emerged as both important and feasible (i.e., within the “go-zone” of each cluster). The metacategories, clusters, and above-average importance and above-average feasibility (i.e., upper right quadrant of the bivariate scatterplot for each cluster) items are shown in to . The pattern match () revealed substantial correspondence between the patterns of ratings of importance and feasibility, both graphically and statistically. Minor variation in the order of the clusters based on average cluster rating of importance and feasibility was observed, and the similarity in the two patterns was confirmed by a correlation coefficient of r = .87.
Table 3. Clusters and statements related to Clinic, Community and Culture, rated above cluster average for both Importance (Imp) and Feasibility (Feas).
Table 4. Clusters and statements related to Health System Understanding, rated above cluster average for both Importance (Imp) and Feasibility (Feas).
Table 5. Clusters and statements related to Population Health Science & Data, rated above cluster average for both Importance (Imp) and Feasibility (Feas).
Figure 3. Pattern match of competency cluster ratings of importance and feasibility.
As a final interpretive step, the clusters and metacategories identified via the current study were mapped to six other sets of competencies produced by professional associations and other studies. summarizes the comparison. Although the grouping differed, we found that there was extensive overlap between topics and content subsets reflected in our results, and competencies/content/domains covered by the other published lists, with minor exceptions.
Table 6. Concept-mapping competencies comparison.
Insights
A century after Flexner defined the “modern” medical school curriculum, an update is needed that better aligns educational, research, and clinical missions with the needs of the population.Citation34,Citation35 Progress toward this goal has included repeated calls to enhance the integration of PH instruction into medical education. Although multiple bodies have laid out broad frameworks for this integration, and in some cases specified detailed content, a gap still remains in defining what PH content is most feasible and important to introduce into a 21st-century medical school curriculum.
We generated 116 competencies divided into 12 clusters and three metacategories. Among clusters, clinical prevention emerged with the highest score when both importance and feasibility were considered (). This may not be surprising given the makeup of our competencies work group, as this is an integral part of FM training and practice, one that academic family physicians and physicians from other specialties are often tasked with teaching. The lower feasibility ratings for the additional clusters deemed to be most important (cultural competence, social determinants of health, and community engagement) may point to the need for engagement with different types of faculty and stakeholders outside of the traditional life science and clinical experts found in most medical schools, in medical centers, and in hospital settings. Greater incorporation of PH and social scientists, as well as community leaders and members of diverse patient populations, as educators and partners in the general medical curricula may be a long overdue to address these issues. The lower feasibility ratings of “community engagement” and “social determinants of health” may also point to a perception among our respondents of barriers to achieving such integration and to the need for developing a deeper pool of creative curricular solutions to providing this content outside of the exam room. Overall, little difference was found between the pattern of ratings of importance and feasibility, suggesting convergence around the PH instruction content that should and can be integrated in medical education, especially for those higher rated clusters.
The comparison of our competencies, clusters, and megadomains to other published competency lists—and similarities found therein—gives us confidence that the competencies identified by our concept-mapping participants have external validity and that it does not veer much from well-established lists. This comparison also highlighted the problem of terminology and definitions, with each author or organization using different terms to describe content, some using “domains,” others “competencies,” “content,” or “components,” making a direct comparison somewhat challenging.
The strengths of this study include multi-institutional collaboration and the inclusive process that incorporated the perspectives of both FM educators and PH professionals. The study was not designed to identify differences between PH professionals or FM educators’ responses. Many participants were dually trained (). This sampling frame supports face validity for FM educators seeking to incorporate PH content into their curriculum.
Our results may be limited by convenience sampling, so generalizability to all FM educators is uncertain. In addition, generalizability to medical educators from other specialties and professions is difficult to determine. There may be an inherent bias due to the group of participants who, by virtue of their specialty and their willingness to complete the process, are likely already convinced of the importance of incorporating PH competencies into medical education. This group's mind-set is likely not representative of the wider community of medical educators and leaders. Furthermore, and perhaps even more important, although competencies were prioritized by knowledgeable educators, patients and communities were not included in this initial round of prioritization, so these conclusions cannot be assumed to reflect the priorities of patients and communities.
An additional limitation of the study was the open-ended structure of the competency statement prompt. Many statements were not worded using competency language and will need to be reworded, reframed, and/or reinterpreted by individuals and institutions. Guidance for this process is available from many organizations including the Institute of Medicine/National Academy of Medicine, Association for Prevention Teaching and Research, AAMC, and Centers for Disease Control and Prevention. An additional methodological limitation included the subjectivity introduced by the research team's (a) “idea synthesis” (condensing, editing, amendment, modification, and de-duplication of submitted statements) and (b) selection and titling of the most meaningful cluster solution during “data analysis.” Finally, this study probably significantly undersampled representatives from osteopathic institutions, an important consideration given that outcomes on the graduate surveys administered by the American Association of Colleges of Osteopathic Medicine indicate much higher levels of PH instruction occurs in that context.Citation36
Further inquiry building on this research could follow several paths. First, we must include a broader representation of educators across medical specialties and health professions. Second, we should engage medical students, residents, patients, and community members. Their inclusion would help to increase the broad validity of curricular priorities and would help fulfill a moral and social obligation. Third, it is also important to work with educators across the various stages of medical education to clarify when is most appropriate time to deliver content and help guide updates to accreditation standards. It would be ideal to incorporate PH competencies into the AAMC Entrustable Professional Activities. Finally, connecting an emerging published and gray literature on curricular innovations to specific competencies would help draw a detailed map for institutions new to incorporating this content.
Compared to conventional qualitative or quantitative approaches alone, group concept mapping provides an alternative structured and unified approach to data analysis and representation. It facilitates a participatory-based research approach that intentionally includes representatives of the target population in multiple phases of the research process.Citation21 The visual representations generated by the concept-mapping software may enhance our ability to communicate the complex relationships among the data points and facilitate an understanding of cues that are not readily apparent when presented as lists or tables, especially with regard to value-based input like feasibility and importance.
Given the broad scope of PH and the reality that, with additional time, expense, and difficulty, one can obtain a master's degree in PH or a related discipline in 1 year, we propose that priority content for inclusion in a medical school curriculum should be identified and should occupy less than 1 year of curricular time in total, either as a block or as longitudinally integrated curricula.
Finally, FM as a specialty is well positioned to lead the integration of PH into medical school curricula. The American Academy of Family Physicians supports the integration of primary care and PH,Citation37 academic family physicians often serve in leadership roles in preclinical doctoring courses, FM clerkships are distributed in many neighborhoods and communities, and practicing family physicians provide clinical preventive care. In addition, beyond medical school, FM residency programs are often located in community settings.
In conclusion, we have identified and organized a set of topics that serve as a foundation for the integration of FM and PH education. Incorporating these topics into medical education is viewed as important and feasible by FM educators and PH professions. Future steps include the development, refinement, and integration of these topics into competency language, milestones, and entrustable professional activities to help address core PH competencies in medical education.
References
- Institute of Medicine. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century (K Gebbie, L Rosenstock, LM Hernandez, eds.). Washington, DC: The National Academies Press, 2003. http://www.nap.edu/openbook.php?record_id=10542. Accessed November 18, 2016.
- Institute of Medicine. Training Physicians for Public Health Careers (K Gebbie, L Rosenstock, LM Hernandez, eds.). Washington, DC: The National Academies Press, 2007. http://www.nap.edu/catalog.php?record_id=11915. Accessed March 20, 2013.
- Zenzano T, Allan JD, Bigley MB, Bushardt RL, Garr DR, Johnson K, et al. The roles of healthcare professionals in implementing clinical prevention and population health. American Journal of Preventive Medicine 2011;40:261–7.
- Chamberlain LJ, Wang NE, Ho ET, Banchoff AW, Braddock CH, Gesundheit N. Integrating collaborative population health projects into a medical student curriculum at Stanford. Academic Medicine 2008;83:338–44.
- Finkelstein JA, McMahon GT, Peters A, Cadigan R, Biddinger P, Simon SR. Teaching population health as a basic science at Harvard Medical School. Academic Medicine 2008;83:332–7.
- Maeshiro R. Responding to the challenge: population health education for physicians. Academic Medicine 2008;83:319–20.
- Ornt DB, Aron DC, King NB, Clementz LM, Frank S, Wolpaw T, et al. Population medicine in a curricular revision at Case Western Reserve. Academic Medicine 2008;83:327–31.
- Michener JL, Yaggy S, Lyn M, Warburton S, Champagne M, Black M, et al. Improving the health of the community: Duke's experience with community engagement. Academic Medicine 2008;83;4:408–13.
- Kerkering KW, Novick LF. An enhancement strategy for integration of population health into medical school education: employing the framework developed by the Healthy People Curriculum Task Force. Academic Medicine 2008;83:345–51.
- Association for Prevention Teaching and Research (Healthy People Curriculum Task Force). Clinical Prevention and Population Health Curriculum Framework—Version 3. Clinical Prevention And Population Health Curriculum Framework, 2015.
- Association for Prevention Teaching and Research. 2013. Advancing Interprofessional Clinical Prevention and Population Health Education: Curriculum Development Guide for Health Professions Faculty. https://aptr.site-ym.com/?crosswalk. Accessed July 6, 2015.
- Kaprielian VS, Silberberg M, McDonald MA, Koo D, Hull SK, Murphy G, et al. Teaching population health: A competency map approach to education. Academic Medicine. 2013;88:626–37.
- Michener JL, Koo D, Andolsek K, Murphy G. ACGME: Population Health Curriculum Integration Materials | Practical Playbook. http://www.practicalplaybook.org/. Accessed July 6, 2015.
- Association of Schools & Programs of Public Health. Population health across all professions. http://www.aspph.org/ftf-reports/population-health-in-all-professions/. Accessed July 6, 2015.
- Association of American Medical Colleges. Medical School Graduate Questionnaire 2014. All Schools Summary Report. https://www.aamc.org/download/397432/data/2014gqallschoolssummaryreport.pdf. Accessed June 29, 2015.
- Association of American Medical Colleges. Medical School Graduate Questionnaire 2011. All Schools Summary Report. https://www.aamc.org/download/263712/data/gq-2011.pdf. Accessed March 20, 2013.
- Trochim WMK. An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning 1989;12:1–16.
- Kane M, Trochim WMK. Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage, 2007.
- Baldwin CM, Kroesen K, Trochim WM, Bell IR. Complementary and conventional medicine: A concept map. BMC Complementary and Alternative Medicine 2004;4:2.
- Rosas SR, Camphausen LC. The use of concept mapping for scale development and validation in evaluation. Evaluation and Program Planning 2007;30:125–35.
- Rosas SR, Kane M. Quality and rigor of the concept mapping methodology: A pooled study analysis. Evaluation and Program Planning. 2012;35:236–45.
- Burke JG, O'Campo P, Peak GL, Gielen AC, McDonnell KA, Trochim WMK. An introduction to concept mapping as a participatory public health research method. Qualitative Health Research 2005;15:1392–1410.
- Concept Systems Incorporated. The Concept System® Global MAX™ Build 2013.322.11. Web-based application. 2012. Available from www.conceptsystems.com.
- Krippendorf K. Content analysis: An introduction to its methodology. 2nd ed. Thousand Oaks, CA: Sage, 2004.
- Kruskal JB, Wish M. Multidimensional scaling, Issue 11. Thousand Oaks, CA: Sage, 1978.
- Kruskal JB. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 1964;29;1:1–27.
- Everitt B. Cluster analysis. New York, NY: John Wiley; 1980.
- Association for Prevention Teaching and Research. Clinical Prevention & Population Health Curriculum Framework. http://www.teachpopulationhealth.org/ Accessed on November 18, 2016.
- Maeshiro R, Johnson I, Koo D, Parboosingh J, Carney JK, Gesundheit N, et al. Medical education for a healthier population: Reflections on the Flexner Report from a public health perspective. Academic Medicine 2010;85:211–9.
- Harris D, Puskarz K, Golab C. Population health: Curriculum framework for an emerging discipline. Population Health Managment 2016;19:39–45.
- Catalanotti JS, Popiel DK, Duwell MM, Price JH, Miles JC. Public health training in internal medicine residency programs: A national survey. American Journal of Preventive Medicine 2014;47:S360–7.
- Trochim WMK. The reliability of concept mapping. Paper presented at: The annual meeting of the American Evaluation Association; November 6, 1993; Dallas, Texas.
- Morley CP, Prunuske J, Mishori R, Rosas SR, Jarris Y, Jordan W. Statements table—Essential public health competencies for medical trainees: Establishing a consensus in family medicine. STFM Resource Library. http://fmdrl.org/5654. Accessed June 22, 2015.
- Kaufman A, Roth PB, Larson RS, Ridenour N, Welage LS, Romero-Leggott V, et al. Vision 2020 measures University of New Mexico's success by health of its state. American Journal of Preventive Medicine 2015;48:108–15.
- Morley CP, Mader EM, Smilnak T, Bazemore A, Petterson S, Rodríguez JE, et al. The social mission in medical school mission statements: Associations with graduate outcomes. Family Medicine 2015;47:427–34.
- AACOM 2014–15 Academic Year Graduating Seniors Survey. 2016.
- American Academy of Family Physicians. Integration of primary care and public health (Position paper). http://www.aafp.org/about/policies/all/integprimarycareandpublichealth.html. Accessed February 9, 2016.