A randomized pilot study of the use of concept maps to enhance problem-based learning among first-year medical students

Abstract

Background: Medical student education is challenging, and concept maps (CMs) can help students link new and existing knowledge, promote critical thinking and identify knowledge gaps.

Aims: To study the feasibility, acceptability, and effectiveness of CMs in problem-based learning (PBL) tutorials.

Method: Students and tutors were randomized to tutorials that used or did not use CMs. A mixed-methods approach generated qualitative and quantitative results of participants’ perspectives on and the effects of CMs in PBL tutorials.

Results: Student survey response rate was 71% (122/172). Most students (82.6%) planned to use CMs in the future at least occasionally, and students in CM tutorials endorsed increased likelihood of using CMs in the future (p = 0.02) versus students in non-CM tutorials. Qualitative analyses identified consistent associations between CMs and recurrent themes: integration of physiological mechanisms, challenging students’ knowledge of the material, and identification of knowledge gaps. Quantitative assessment of final exam scores revealed a statistically significant increase in the students’ scores in CM tutorials versus students in non-CM tutorials with an a priori α of <0.10.

Conclusions: CMs are well accepted by students and faculty, feasible to incorporate into PBL tutorials, and may result in improved exam performance and student learning of physiologic concepts.

Introduction

Medicine has experienced exponential growth of biomedical knowledge in the last two decades. To teach medical students to learn, retain, and apply the ever-growing body of biomedical information, new pedagogic methods are needed (Epstein 2004).

In standard lectures, students are passively exposed to factual content, and do not learn or apply concepts. In contrast, in problem-based learning (PBL) tutorials students learn actively using case-based peer-to-peer teaching. In PBL tutorials, students use content from lectures and independent study to approach cases under the guidance of a faculty learning facilitator, referred to as a tutor (Epstein 2004; Knowles et al. 2005). Ideally, the PBL experience should inspire curiosity, enhance personal initiative and allow free expression of learners’ ideas (Dewey 1938).

PBL has become increasingly common in medical schools’ curricula, ostensibly as one possible solution to preparing students to cope with the rapid expansion of biomedical knowledge. PBL has three main learning goals (Belland et al. 2009): (1) to promote deep content learning (Bloom 1956; Belland et al. 2009); (2) to promote problem-solving skills (Scandura 1977; Schoenfeld 1985; Bodner 1991; Glaser et al. 1992), determine relevant resources to help analyze the problem (Schoenfeld 1985), and develop a solution to address the problem (Hmelo-Silver 2004) and (3) to promote self-directed learning (Hmelo et al. 1997).

Table

Practice points

This randomized study demonstrates that the incorporation of CMs in PBL tutorials is both feasible and acceptable to students and tutors. Associations between CMs and recurrent themes included integration of physiological mechanisms, challenging knowledge of the material, and identification of knowledge gaps. A significant increase in final exam scores of students using CMs was noted.

Despite increasing use of PBL, students continue to struggle to integrate new knowledge with prior learning and to apply basic science knowledge to clinical scenarios. Students frequently find it difficult to transfer knowledge learned in one context (e.g. lecture) to a similar problem in a different context (e.g. tutorial or clinical cases). Tutors typically have difficulty diagnosing the learning problems accounting for these challenges. Even when learning issues are identified, instructors struggle with devising effective strategies to help students overcome them.

To address these educational challenges, we designed the current study to assess the effects of concept maps (CMs) on medical and dental students in a first-year physiology course. CMs are graphical tools for organizing and representing knowledge, developed in the 1960s and 1970s by Novak and Gowin (1984) to support learning in the sciences. CMs were originally defined as diagrams that describe the one's understanding of correlations between concepts in a specific area of knowledge. CMs have subsequently been modified and used as visual representations of how knowledge is organized and represented in memory (Torre et al. 2007).

CMs have the potential to impact medical education in several ways. First, CMs allow students to link new knowledge with existing knowledge. In this manner, they may help pre-clinical students integrate basic science with clinical problems. Second, CMs may promote critical thinking and PBL by allowing students to move from linear to integrated, holistic thinking. Third, CMs allow teachers to assess student learning and identify areas of weakness (Boxtel et al. 2002; Torre et al. 2007; Daley & Torre 2010). Observing the structure and detail of CMs, teachers can identify reasoning difficulties, which permits delivery of targeted feedback, clarification of erroneous concepts and adjustment of teaching goals (Edmonson & Smith 1998; Roberts 1999; Pottier et al. 2010). Precise teacher feedback is crucial for improving student problem-solving performance and decreasing students’ knowledge deficits (Morse & Jutras 2008).

Although the value of CMs in medical education has been broadly demonstrated, it is not known whether using CMs in PBL settings is feasible and acceptable to both faculty and students. Furthermore, it is also not known whether using CMs in PBL tutorials can improve student performance on exams or in clinical settings.

In our study of first-year students in a physiology course, students constructed CMs in tutorial sessions to link causative physiological mechanisms to different clinical findings in PBL cases. Designed as an educational intervention pilot study, this study was intended to assess the feasibility, acceptability and preliminary effectiveness of CMs in PBL tutorials. Our study also explored the potential risks of using CMs in PBL tutorial sessions.

Our hypotheses were: (1) teaching physiology to first-year medical students using CMs would be feasible for tutors; (2) use of CMs in the PBL setting would be perceived as acceptable and beneficial to the majority of educators and students; (3) students in CM tutorial groups would demonstrate equivalent or better knowledge of physiology compared to students from non-CM groups and (4) CMs would not be detrimental to students’ learning.

Methods

Study population and setting

All first-year Harvard Medical School (HMS) medical and dental students in the class of 2014 in the New Pathway curriculum enrolled in the Integrated Human Physiology (IHP) course were eligible to participate, as were all small-group instructors involved in the course. Students and instructors received an introductory email containing a study fact sheet and an invitation to participate. The fact sheet explained the purpose of the study, data collection, a confidentiality and anonymity statement and assurance that participation was voluntary with no consequences for non-participation. The study protocol was reviewed and approved by the HMS IRB.

As per course protocol, the course manager (who was not involved with the study) randomly assigned each student to one of 20 tutorial groups (8–9 students per tutorial group). Tutors who agreed to participate were asked whether they preferred to use CMs in their tutorial group or not. To reduce potential bias resulting from tutors’ preferences, tutors who preferred to use CMs were randomly assigned to a CM tutorial (Group A) or a non-CM tutorial (Group C). Tutors expressing no preference were assigned to non-CM tutorials (Group B). Students assigned to tutors who did not participate in randomization were grouped together (Group D; Figure 1). There were five cases (each comprised of three tutorial sessions) during the course.

Figure 1. Flow chart of the determination of the teaching method (CM or non-CM) for each tutorial group.

All tutors in the IHP course, regardless of group assignment or study participation, received the same faculty development training on CMs. Tutors read two papers on CMs (Guerrero 2001; Torre et al. 2007) and attended a didactic session run by the course director (RMS) in which the purpose and development of CMs were reviewed. Specifically, the course director demonstrated how a CM could be constructed using the course's first tutorial case and provided examples of completed CMs.

Of note, all students, regardless of tutorial assignment, had some baseline exposure to CMs as the course director used CMs in four interactive large group sessions involving the entire class during the six-week course.

Data collection

Both quantitative and qualitative data were collected for the study, including a student survey, tutor interviews, and student final exam grades.

Student survey

An anonymous, online, end-of-course survey was developed using REDCap (2011), a secure, web-based survey application (Harris et al. 2009; REDCap Survey, Nashville, TN). The survey assessed students’ attitudes and experiences regarding the use of CMs, and whether they planned to continue, start or stop using CMs after the course and why. A draft of the survey was administered to 165 second-year HMS students (HMS class of 2013), who provided feedback on the content and format of the survey questions, specifically with regard to clarity and readability of the questions. The survey was revised based on their feedback. Rigorous assessment of reliability and validity of the survey could not be performed due to lack of a gold standard assessment tool regarding students’ impressions of using CMs as educational tools. A link to the final secure survey was sent to first year students, with reminders that participation was voluntary and that no identifiers would be collected.

Tutor interview

Tutors in Groups A, B and C were asked to participate in individual 20-minute audio-taped, semi-structured interviews conducted by one of the investigators (CV) after the course had ended. Interviews assessed tutors’ attitudes regarding CMs and their opinions of benefits and drawbacks of using CMs in the PBL tutorials. Interviews were transcribed and de-identified for analysis.

Final exam grades

Students’ final exam grades were collected, de-identified, and linked to student tutorial groups by a co-investigator (AMS) not involved in the IHP course or student evaluation. The final exam comprised a series of short essay questions focused on the physiological mechanisms associated with two clinical cases. The cases alone were given to the all students enrolled in the course 48 hours in advance of the written examination to allow students to analyze the cases and prepare for the exam. Exams were graded without knowledge of whether students were assigned to tutorials using CMs.

Data analysis

Student survey

Student survey data were imported to SAS version 9.2 for statistical analysis of closed-ended items. We performed chi-square tests of independence to compare student responses across tutor groups. Open-ended questions about benefits and drawbacks of CM were coded for major themes by one investigator (CV), with subsequent rounds of iterative re-coding and analyses with co-investigators.

Tutor interview

Tutor interview transcripts were analyzed using the Framework qualitative data analysis approach (Ritchie & Lewis 2003). Four investigators participated in multiple rounds of reading the transcripts and coding them in order to identify the major themes. Once themes were identified, all transcripts were coded using Weft software (Weft QDA 2011). Final coding carried out by all investigators was compiled into one document after consensus was reached.

Final exam grades

Student final exam scores were analyzed using SAS PROC GLM to compare means across tutorial groups. As our study was designed as a pilot study exploring preliminary evidence of effectiveness of the CM method, we set an a priori significance level of α < 0.10 (two tailed).

Results

Student survey

HMS class of 2014 is composed of 172 medical and dental students, 50% of whom are male. No student declined participation in the study. Our survey response rate was 71% (122/172). Of students in CM tutorials, 52% reported that this was their first experience using CMs. The majority of students in CM tutorials reported that the use of CMs was “almost always” or “frequently” helpful in their learning. Sixteen percent described CMs as “rarely” (13.4%) or “never” (2.4%) useful. Sixty percent of students in CM tutorials said that CMs helped them think critically, and half felt CMs helped them formulate hypotheses about a case “almost always” or “frequently” (Table 1).

Table 1.  Student reports of impact of CMs in tutorial groups (N = 81)

CMs in tutorials helped me:	Almost always (%)	Frequently (%)	Occasionally (%)	Rarely (%)	Never (%)
Think critically about a case	25.6	34.1	22.0	15.9	2.4
Formulate hypotheses about a case	19.5	30.5	32.9	14.6	2.4
Identify areas I did not fully understand	16.1	25.9	32.1	18.5	7.4
Identify areas I understood well	13.4	36.6	39.0	7.3	3.6

Most students (82.6% [101/122]) reported that they planned to use CMs at least occasionally in their future classes. Participation in a CM tutorial group was associated with greater likelihood of future plans to use CM (p = 0.02, Table 2).

Table 2.  Student acceptability of CMs (N = 122)

Q: I plan to use CMs in my future classes:
	Students who used CM (%)	Students who did not use CM (%)	All students (%)
In most classes	10.9	5.3	9.1
In more than a few classes	26.5	7.9	20.7
In a few classes	50.6	57.9	52.9
No	12.0	28.9	17.4

P value for the overall difference between the CM and non-CM groups = 0.02

When asked to list in an open-ended manner how CMs helped them learn, students’ answers provided the following major themes: CMs were useful in determining causality, CMs helped identify areas of weaknesses, CMs enhanced knowledge integration, and CMs facilitated hypothesis generation. When asked to list the ways in which CMs impeded students’ learning of physiology, major themes included: organization issues, time issues, limited utility as a vehicle for review, and intimidating or overwhelming to design and use. Table 3 lists the major themes with representative quotes.

Table 3.  Open-ended survey questions major themes

Major Themes	Illustrative Quotes
Q: Please list any ways in which CMs helped you learn Physiology: (n = 59 responses)
Determine causality	Helped to look at causality and connections between different parts of the clinical presentation.
	They really emphasize cause and effect relationships and how symptoms and pathways are interrelated.
Helped identify weaknesses	They also forced me to challenge my understanding and helped me identify areas that needed more work.
	Showed areas where we were making broad, unjustified leaps.
Helped integrate knowledge	They helped tie together constellations of symptoms, and see how one physiological change can affect multiple organs or cause multiple symptoms.
	It also highlighted linkages that I might not have pieced together myself and helped me learn the integration in a pathologic state across the various systems we had been focusing on each week.
Helped develop hypotheses	It helped to clearly understand the connection between various physiological factors. Helped me generate several hypotheses to explain one outcome. Forced me to state my assumptions and explain them.
	They helped me think of a broad array of causes for a specific symptom, so even if it wasn't right for the case at hand, I could think about it for future cases and see if it fit the picture in those cases.
Q: Please list any ways in which CMs impeded your learning of Physiology: (n = 55 responses)
Organization issues	They were extremely convoluted. Often times there were too much going on and it did not present information in an organized matter. They were not spatially well organized so did not help to remember the information because seemed like random scatterings.
	It's difficult to organize information effectively with CMs, particularly for topics that have lots of detail that require many connections and bubbles. The map ends up very complicated and hard to read.
Time Issues	CMs slowed the pace of tutorial; unnecessarily time-consuming.
Not useful for reviews	CMs are also difficult to come back to later and look at the connections you had previously made.
Intimidating/Overwhelming	At first they were intimidating, but never impeded my learning.

Tutor interview

Forty-five percent (9/20) tutors involved in the IHP class volunteered to be randomized and to participate in the interviews (Groups A, B and C). Seventy-eight percent (7/9) of the tutors interviewed were teaching this course for the first time, while two had taught the course for five years.

The distribution of tutors in the CM versus non-CM groups is depicted in Figure 2. Based on students’ survey answers, we were able to determine that 45% (5/11) of the groups with tutors who did not participate in the study itself (Group D) used CMs during their tutorials (Figure 2).

Figure 2. Summary of tutorial teaching format assignments. Of the 10 tutorials that used CMs, five were Group A and five were Group D. Non-CM tutorials included two were Group B, two were Group C and six were Group D.

Eight of the nine interviewed tutors had positive attitudes toward CMs. One of the tutors in Group C said, “I was really happy to explore [CMs] because the way it was presented and the way it was described to us, I thought it was a brilliant way to teach.” Tutors in Group A reported encountering initial student resistance toward the use of CMs in the tutorial setting; however, they noted that initial resistance gave way to acceptance as the students appreciated the benefits of the new method.

Tutors in all groups identified several benefits of using CMs as a learning tool in tutorials. Tutors most commonly described CMs as enhancing critical thinking, as evidenced by the comment: “I think CMs force you to think about a problem in a rigorous fashion, because by definition you have to diagram and explicitly link mechanisms to symptoms and mechanisms to other mechanisms. By being forced to make those connections you are forced to consider a problem holistically as opposed to just superficially.”

Tutors also felt that CMs made students explore various hypotheses rather than attempt to make a diagnosis too early in the discussion. A tutor from Group A described the tendency of students to prematurely formulate diagnoses, and observed that CMs helped prevent this: “The students pursue going towards a diagnosis too quickly, and then try to tie everything together that is consistent with a diagnosis rather than keeping the analysis more open ended so that physiology can be explored and you don’t restrict yourself to the diagnostic line of questioning.”

Additionally, all tutors felt that CMs provided more structure to tutorial, as CMs helped students to better focus and participate and allowed tutors to better observe students’ thinking and reasoning. As noted by a tutor in Group B, “(CM) is a representation of what the group was thinking, what the uncertainties are, where the difficulties are, where the false conclusions are, and the group consensus.”

When asked about the drawbacks of CMs, tutors in Group A (CM group) cited no prohibitive drawbacks to using CMs. Tutors in Groups B and C (non-CM groups) noted that developing CMs could be time-consuming and that CMs may not fit some students’ learning styles.

Acknowledging the time used to generate CMs, tutors in Group A still felt CMs overall were good learning tools. A tutor from Group A stated, “the time they use to build a CM makes up for the time they save not running around in circles.”

Eight of nine (89%) of the interviewed tutors said they planned to use CMs in their teaching in the future. The one tutor who dissented did so not because she believed they were ineffective but because, in the spirit of PBL tutorials, she planned to leave the decision to use CMs or not up to students.

Final exam grades

We compared mean final exam scores and standard deviations of Group A to Groups B and C. The mean score for the final examination for all students in the IHP course was 156.0 (SD = 12.9, range 119.0–180.5). The maximum score possible on the final exam was 182. Students in Group A tutorials scored slightly higher on the final examination than the students in Group B and C tutorials (mean [SD]: Group A = 157.5 [11.7] versus Groups B/C = 152.3 [13.1]; F-statistic = 3.34, p = 0.07). This difference was statistically significant using our a priori level of significance of α < 0.10.

Discussion

In this study, we demonstrate that using CMs in PBL tutorials was feasible, acceptable to students and tutors, and may enhance learning and exam performance in a pre-clinical first-year physiology course. Tutors with varying levels of experience in the course reported they were able to integrate CMs into their tutorials. Student survey responses and tutor interviews demonstrate broad acceptability of using CMs in tutorial for both students and tutors. Exam scores demonstrated modestly better performance by students in CM groups as compared to students in non-CM groups, suggesting that students exposed to CMs may have developed better understanding of concepts and/or been more able to effectively apply information to solving case-based problems.

To create a complete CM, students must translate the information they learned from textbooks and lectures into a network of knowledge that has meaning, value and recognized utility (Hendricson et al. 2006). CMs also encourage exploration of different physiological hypotheses before formulating a diagnosis, possibly increasing diagnostic accuracy and reducing cognitive errors from premature closure (Auclair 2007; Coderre et al. 2010; Eva et al. 2010). Considering and ruling out alternative and incorrect physiologic explanations while building a CM may improve students’ future performance if similar techniques are applied to actual patients and cognitive errors, such as premature closure and anchoring, are avoided (Crokerry 2003). Furthermore, clinicians may be more likely to commit errors in their careers if fewer errors are committed during their education (Eva 2009). As such, considering and discarding potential mechanistic explanations in building a CM serves to reinforce physiologic relationships as well as potentially improve future clinical reasoning.

In creating a CM, students are also able to identify gaps in their knowledge and understand the need to seek clarification. In our study, CMs complemented the PBL tutorial environment, as the collaborative nature of information gathering, hypothesis generation and identification of learning issues in PBL directly informed the construction of and were directly visualized in students’ CMs (Rendas et al. 2006). To the extent that CMs provided tutors with a greater appreciation of errors in student thinking, feedback was enhanced. According to Schiff, the absence of feedback is often identified as a reason for the proliferation of diagnostic errors (Schiff 2008).

Tutor interviews revealed how CMs helped students understand causality: instead of going from a sign or symptom (point A) to a conclusion or diagnosis (point C), developing CMs prompted students to acknowledge mechanistic relationships (and go from point A to B1 to B2 to B3 before arriving at point C). CMs helped students integrate physiological mechanisms, challenged their knowledge of the material, and allowed them to identify gaps in their knowledge. Most tutors agreed that CMs enhanced students’ critical thinking, provided more structure to tutorial, and slowed the pace of tutorial, which allowed students to better focus and participate. Tutors in CM tutorials felt they could better observe students’ thinking, better identify misconceptions, and thereby provide students with more accurate feedback.

The modestly higher final examination scores of students in CM tutorials may reflect improved learning of core physiologic concepts. In support of this possibility, both tutors and students agreed that CMs pushed students to explore different hypotheses before generating an explanation for the findings in the case. If students who used CMs explored different physiological hypotheses when approaching a problem, then perhaps they were less prone to jump to an answer or try to fit the available information to pre-formulated conclusions. However, it is important to emphasize that performance on the final exam was only one manner of assessing the impact of CMs, and it is not meant to be viewed as a gold standard. Both tutor interviews and student surveys highlighted important aspects to support the efficacy of CMs, such as enhancement of critical thinking, investigation of multiple hypotheses rather than engaging in premature closure, and self-identification of gaps of knowledge.

There were drawbacks of CMs reported by tutors and students. Students’ main concerns with CMs related to difficulties in spatially organizing maps, which sometimes resulted in convoluted final products that were difficult to interpret and review. Although creating CMs was occasionally described as time consuming, the time and effort used to construct CMs may have resulted in less time expended on tangential discussions. Additionally, while some tutors appreciated student resistance to using of CMs early in the course, resistance faded as the course progressed and students became more comfortable using CMs. As such, more explicit and thorough introduction to CMs earlier in medical school may further increase students’ acceptability of CMs.

There are several limitations associated with our study. Our study included only one physiology course in the first year medical school curriculum; it is not known whether CMs would yield similar results in other courses. In addition, this study explored preliminary effectiveness of CMs using end-of-course grades. More targeted outcomes, such as measures of critical thinking and longitudinal measures of clinical reasoning are needed to assess more specific effects of CMs. Future work will include assessing the long-term effects of CMs on clinical performance, such as preventing premature closure and reducing misdiagnoses.

Importantly, variation in tutor understanding of CMs suggests that future studies should employ more in-depth faculty development to achieve better understanding of and fidelity to this intervention. Future work will include development of more structured and rigorous faculty development curricula to optimize (and normalize) faculty understanding of and comfort with using CMs.

Finally, because we designed our study as a pilot study, we set a liberal α level of 0.10 to explore the potential effectiveness of CMs on exam performance; on the other hand, the fact that all students were exposed to CMs in interactive lecture sessions could have biased the study toward the null hypothesis. Future phase II trials assessing CMs in PBL tutorials may need a more stringent α level to detect between-group differences.

Our study had several strengths that support the validity of our findings. Multiple methods of assessment of multiple levels of respondents (students and tutors) coupled with consistency of findings across survey, interview, and final exam results support the validity of inferences drawn from these data. In addition, the high student survey response rate suggests decreased influence of non-response bias on survey results.

Conclusion

We conclude that CMs are well accepted by students and faculty, are easy to incorporate into PBL tutorial sessions, and may result in improved exam performance and student learning of physiologic concepts. Furthermore, CMs may improve students’ performance by beneficially affecting the way students organize knowledge and approach problems. CMs enable instructors to better observe students’ thought processes and identify areas of weakness or misunderstanding. The use of CMs in PBL tutorials may help stimulate active, deep learning of basic sciences, better retention of knowledge content, and better application of this knowledge to novel problems. As such, CMs may be a tool to help bridge the gap between basic science and clinical practice. Expanded use of CMs in pre-clinical courses may help students appreciate the importance of exploring multiple hypotheses before deciding on a diagnosis, and reduce cognitive errors and misdiagnosis.

Declarations of interest: The authors have no declarations of interest to report.

Carolina Veronese was supported by the Robert Leffert Scholar in Medical Education grant through HMS.