The impact of deliberate reflection with WISE-MD™ modules on surgical clerkship students’ critical thinking: a prospective, randomized controlled pilot study

Abstract

Purpose

Critical thinking underlies several Association of American Medical Colleges (AAMC)-defined core entrustable professional activities (EPAs). Critical-thinking ability affects health care quality and safety. Tested tools to teach, assess, improve, and nurture good critical-thinking skills are needed. This prospective randomized controlled pilot study evaluated the addition of deliberate reflection (DR), guidance with Web Initiative in Surgical Education (WISE-MD™) modules, to promote surgical clerks’ critical-thinking ability. The goal was to promote the application of reflective awareness principles to enhance learning outcomes and critical thinking about the module content.

Participants and methods

Surgical clerkship (SC) students were recruited from two different blocks and randomly assigned to a control or intervention group. The intervention group was asked to record responses using a DR guide as they viewed two selected WISE-MD™ modules while the control group was asked to view two modules recording free thought. We hypothesized that the intervention group would show a significantly greater pre- to postintervention increase in critical-thinking ability than students in the control group.

Results

Neither group showed a difference in pre- and posttest free-thought critical-thinking outcomes; however, the intervention group verbalized more thoughtful clinical reasoning during the intervention.

Conclusion

Despite an unsupported hypothesis, this study provides a forum for discussion in medical education. It took a sponsored tool in surgical education (WISE-MD™) and posed the toughest evaluation criteria of an educational intervention; does it affect the way we think? and not just what we learn, but how we learn it? The answer is significant and will require more resources before we arrive at a definitive answer.

Keywords:

• simulation
• clinical reasoning
• medical education strategies

Introduction

Critical thinking underlies at least three of the Association of American Medical Colleges’ (AAMC) 13-core entrustable professional activities (EPA) for entering residency.¹ Critical thinking is required for physicians to competently and independently provide patient care.² While critical-thinking ability is clearly related to quality and safety in health care,³ defining and measuring it continue to be a challenge for health professions’ educators, including medical faculty.^4–7

Defining critical thinking has been elusive for most of the recent century.⁸ There is no consensus for an approved definition in the medical literature,⁹ nor is there agreement on terminology to define the process and little evidence for best practices for teaching, measuring, and evaluating critical thinking.^4–6 Others question whether critical thinking can even be taught.⁴ What it means to think critically may vary by discipline, practice settings, and contexts.¹⁰ Critical thinking may be viewed as a variety of ways to think with various styles of reasoning,¹¹ and in the health sciences’ literature, critical thinking is often used interchangeably with clinical thinking, clinical reasoning, and diagnostic reasoning.¹⁰

Because critical thinking is largely conceptual, measurement must be inferred from observable behaviors.⁶ Educational strategies to reveal actual thought processes may include a standardized list of questions – necessitating verbal or written evidence for analysis as a requirement to enhance metacognition and make visible a student’s thought processing.⁴ Huang et al⁴ reported the following strategies for teaching critical thinking: 1) slowing down the pace of the learning process to enable students to digest and apply knowledge, 2) actively engaging the learner in tasks that require problems to be solved, 3) compelling students to justify how they arrived at decisions, 4) making thinking explicit, and 5) requiring self-reflection on the part of the learner.

Given the literature and the above noted gaps, the authors wanted to test the integration of deliberate reflection (DR) with Web Initiative in Surgical Education (WISE-MD™) modules as a means to increase critical-thinking ability. Due to the timing of courses, semesters, and per the protocol submitted and approved as exempt by the institutional review board, the methodology was first tested with nurse practitioner students in an advanced health assessment course³⁹ followed by implementation with medical students during their surgical clerkship (SC). This study evaluated critical-thinking outcomes of SC students by adding metacognitive DR guidance to the learning strategy with WISE-MD™ simulation modules. The authors hypothesized that SC students in the intervention group who were exposed to the DR guide would show a significantly greater pre- to postintervention increase in critical-thinking ability than students in the control group who had no DR guidance.³⁹ The next section provides additional details regarding the development of the methodology.

Rationale for the design strategies developed for this study

WISE-MD™

WISE-MD™ is a series of 35 case-based online teaching modules developed to fill in the gaps in surgical education created by shorter hospital stays along with more of the pre-and postoperative care occurring in outpatient services.^12,13 The American College of Surgeons and the Association of Surgical Education endorsed the WISE-MD™ modules, which were designed to develop clinical reasoning in medical students while seeking consistent, high-quality learning environments to ensure clinical competence.^12,13,39 Among the module topics are those particularly germane to the SC such as appendicitis, breast cancer, gall bladder disease, thyroid disease, and hernias.^12,13 The modules were created for independent study illustrated with video and animation using best practices for multimedia design.^12,13,39

Lasting ~1 hour, each module is introduced with a “fundamentals” section and depicts the patient’s experiences and interactions with the physician from initial presentation, history taking, physical examination, laboratory tests and radiological imaging to preoperative preparation, surgery, and recovery.^12,13,39 Professionalism and communication are emphasized throughout each module, which also includes a summary and key findings from the case.^12,13 The surgical procedure is presented with a graphic depiction alongside the actual surgery process overview, which is very helpful for medical student visualization of the virtual along with the actual surgery.³⁹ For the remainder of the document, the authors refer to the WISE-MD™ modules as the WISE modules.³⁹

Although the WISE modules have been used in medical education since 1998 and are used by >200 medical schools nationally and internationally,^12,13 little has been published regarding their use with medical students. One study found that medical students who viewed the WISE modules trended toward better knowledge and clinical reasoning than students who did not view the modules.^14,39

Reflective practice

To be able to think critically, students must learn to routinely and critically examine their own thinking.^6,15 Requiring medical students to examine their own thinking through thoughtful reflection is an important component of medical education^16–18 as reflection requires the ability to think critically.¹⁹ Reflective practice also promotes professional identity transformation from medical student to physician¹⁶ and seeks to improve diagnostic accuracy.^20,21

The Accreditation Council for Graduate Medical Education (ACGME) core competencies and milestones²² and the AAMC’s Physician Competency Reference Set (PCRS)²³ require physician trainees to reflect upon and analyze practice experiences.¹⁷ Reflective practice can be fostered/enhanced when medical students perceive the case/situation as real, there is some conflict, critical questions are raised, and there is a structured process for reflecting.²⁴ Guided reflection is considered a key element of professional identity formation from medical student to physician¹⁶ with an interactional aspect proposed to develop that identity.²⁵

Furthermore, clinical reasoning may be enhanced by the process of “think aloud”, which occurs when a student verbalizes his or her thoughts while doing an assignment.²⁶ Siddiqui²⁷ found think aloud to be valuable in identifying medical students’ critical-thinking strengths and weaknesses during their ICU rotation. Thus, given the potential value of both reflection and think aloud to make critical thinking overt and for the purposes of this study, author MQ created the term DR to reinforce the notion that the reflective thinking process is an overt skill that compliments the skill set of “deliberate practice” defined by Ericsson.²⁸ DR as a metacognitive learning innovation is tested in this study.

DR

The DR process is introduced for a number of reasons. It implies that the reflective process must be made overt to enhance learning³⁹ as suggested by Croskerry who defines “cognitive forcing strategies” as a means of de-biasing and preventing diagnostic error.^26,39 Although similar to the metacognitive strategy known as self-explanation, DR differs in that it is not restricted to inferences, clarifications, justifications, or monitoring of behavior as is inherent in the definitions of self-explanation.^29–31,39 Rather, DR includes the integration of previous experiences with current experiences and the application of strategic knowledge about self and learning including awareness of affective components such as confidence.⁴⁰ DR incorporates mental representation (selective encoding, combination, and comparison) (A Kalet, New York University, email communication, May 2011.)^30,39 and considers the temporal aspects of reflection – before action, during action, and after action.^32,39 Finally, DR includes a think-aloud or “verbal report” strategy that has been used in debriefing and other thought process research strategies.^6,39 Believing that the value of learning through simulation lies in debriefing and reflection on the simulation experience³³ and that structured reflection improves learning outcomes,^{28,31,34–37,39} the authors reasoned that without specific, systematic instructions for learner reflection or self-debriefing, some of the educational value of the video-based simulation in the WISE modules might be lost.³⁹ The authors further speculated that the personalized, real-time, self-debriefing/reflective component of DR for the SC students might improve the WISE module learning experience and outcomes related to working memory and critical thinking.³⁹ Therefore, for this study, the authors developed specific DR instructions to guide the SC learner to apply the principles of reflective awareness to surgical content in the WISE-MD™ modules with the goal of promoting learning³⁸ and enhancing critical-thinking and learning outcomes.³⁹

Participants and methods

Participants

We recruited the participants from two different blocks of SC students at a New England medical school during the fall semester. The study was presented to students during the first week of their rotation. Thirty-one (72%) of the 43 students volunteered to participate and were randomly assigned to either the control group (n=16) or the intervention group (n=15). Written informed consent was obtained from the participants.

Participants were given access to all the WISE modules and provided with a digital recorder to record their thoughts as they completed several preselected modules: abdominal aortic aneurysm (AAA), cholecystitis, appendicitis, and thyroid nodule.³⁹ Participants received no compensation, viewed the modules, and completed assigned activities on their own time outside of class. Although participation would not influence their grade, SC students were told that the WISE modules might be seen as an advantage in terms of their overall learning.³⁹

Procedure

As outlined in the WISE study flowchart in Figure 1, there were four steps to the procedure for both the control and intervention groups ranging from pretest to posttest.³⁹

Figure 1 WISE-MD™ study flowchart.

Note: Reproduced from Terrien JM, Hale JF, Cahan M, Quirk M, Sullivan K, Lewis J. The impact of deliberate reflection with WISE-MD™ modules on critical thinking of nurse practitioner students: a prospective, randomized controlled pilot study. J Nurs Educ Pract. 2016;6(1):55.³⁹
Abbreviations: AAA, abdominal aortic aneurysm; SC, surgical clerkship; WISE-MD™, Web Initiative in Surgical Education.

Step 1/pretest

The intervention and control groups were divided into two subgroups as close to equal size as possible. One subgroup viewed the AAA module, and the other subgroup viewed the cholecystitis module.³⁹ Each student was provided with a digital recorder and asked to freely record their thoughts while viewing each module.³⁹ For this exercise, all students (both intervention and control) were provided with a “free-thought” guide requesting them to record out loud whatever happens to come across their mind at least three times while viewing the modules.

Step 2

The control group reviewed the thyroid module without using a digital recorder.³⁹ While viewing the same module, the intervention group used a digital recorder to answer questions from the DR think-aloud group instructions, which can be found in Table 1.³⁹ These instructions asked the intervention group to complete the specific DR exercises at specific time points.³⁹

Table 1 Deliberate reflection think-aloud instructions for thyroid nodule module

	Check Box
(1) Before the thyroid nodule module begins, press record and respond to the following questions:
1a. What are your initial thoughts on patients with thyroid nodule?
1b. What kind of experience have you had with patients with thyroid nodules?
1c. How confident are you in working with patients with thyroid nodules (1–10)?
1d. How informed are you about the subject matter (1–10)
(2) Before viewing the history section of the module, press record and respond to the following questions:
2a. What will you be looking for in the history – what will help you differentiate?
2b. What characteristics of the complaint or problem are most important to be clear about and why?
2c. What are some possible pitfalls in the communication that you want to avoid?
2d. Are there assumptions or biases in your thinking that you want to avoid?
(3) After viewing the history section of the module, press record and respond to the following questions:
3a. What was the most important finding and what does it mean?
3b. What strategies did the provider use to achieve his or her goals?
3c. What else could the diagnosis be and why?
(4) Before viewing the PE section of the module, press record and respond to the following questions:
4a. What signs will you be looking for and why?
4b. How will the PE help understand your history findings?
(5) After viewing the PE section of the module, press record and respond to the following questions:
5a. What signs confirmed the diagnosis and why?
5b. How do the PE and history (HX) findings fit together?
5c. Are there any other explanations for the findings?
(6) Before viewing the lab and images section of the module, press record and respond to the following question:
6a. What are the most important tests to order and why?
(7) Before viewing the decision-making section of the module, press record and respond to the following questions:
7a. What do you think you would say to the patient knowing what you know now about the case? What possible treatment
methods would you consider?
7b. How did you arrive at this conclusion?
7c. How do you think the patient is feeling about his or her treatment?
7d. How would you communicate to him or her?
(8) After viewing the surgery section of the module, press record and respond to the following questions:
8a. Having seen the surgery, what complications if any might you now anticipate?
8b. Having seen the surgery, how would you explain the symptoms of pain back to the patient?
8c. Having seen the surgery, what if anything might you have done differently in the history and PE?
(9) After finishing the thyroid module, press record and respond to the following questions on a scale of 1 (least confident/informed)
to 10 (most confident/informed):
9a. How confident are you in working with patients presenting with thyroid nodules?
9b. How informed do you feel now about managing a patient presenting with a thyroid nodule?

Note: Adapted from Terrien JM, Hale JF, Cahan M, Quirk M, Sullivan K, Lewis J. The impact of deliberate reflection with WISE-MD™ modules on critical thinking of nurse practitioner students: a prospective, randomized controlled pilot study. J Nurs Educ Pract. 2016;6(1):55.³⁹

Abbreviation: PE, physical examination.

Step 3

The two groups followed the same procedures as described in step 2 for a second module, appendicitis.³⁹

Step 4/posttest

The two modules (AAA and cholecystitis) used in the pretest were again used in the posttest but switched.³⁹ Using the same free-thought guide as described in step 1, students in both the control and intervention groups were asked to record their thoughts freely while viewing the module.³⁹ Students submitted their digital recorders to the study coordinator (KS) for analysis once they completed the study.³⁹ All 31 study participants were sent a brief follow-up anonymous survey about their experience as a study participant, with the WISE modules.³⁹

Data analysis

Data from students’ digital recordings were transcribed into word documents and imported into NVIVO™ Version 10.³⁹ All authors iteratively coded the transcripts. They reviewed the critical-thinking literature for potential categories to reach con sensus on the final critical-thinking coding. Further description of the development of the categories can be found in the work of Terrien et al.³⁹ Final analysis determined five categories with 10 subcategories, which can be found in Table 2.³⁹

Table 2 Critical-thinking categories, subcategories, and explanations

Category	Subcategory	Explanation
Takes perspective	None	Considering the patient’s, doctor’s, and nurse’s perspectives
Considers alternatives	None	What else could it be? Have I considered other options? Have I been thorough?
Makes associations	Describes	Simply what I am seeing? What does it look like?
	Compares	What are the differences between helpful and non-helpful evidence? Taking things apart to rule in or rule out.
	Prioritizes/evaluates	What evidence/information (visual, verbal, and so on) is important? Prioritize evidence.
		What is important with or without the explanation/rationale?
		What is not important with/without explanation/rationale (justifying certain actions)?
	Integrates	How do data from multiple sources (history, epidemiology, PE, labs and images) fit together – confirming or refuting? How am I putting things together and synthesizing, eg, how do various thoughts/concepts fit with each other?
Anticipates outcomes	Examines assumptions	What was I taking for granted? Was I presupposing anything? Are there ways I usually think about this that are not helpful?
	Predicts outcomes	What will I be looking for? What does the future hold? Statements made with or without underlying reasoning or information used for prediction.
	Considers pitfalls	What negative outcomes do I need to watch out for? What are the possible “pitfalls” or dangers in thinking and/or communicating? What do not I want to miss (eg, pertinent negatives)?
Self-assesses thinking process	Assesses self-confidence	How sure am I? What are my reservations?
	Considers experience	What have I seen and done in the past?
	Evaluates learning style	Considers learning style that works best or is preferred and what does not work, and so on

Note: Reproduced from Terrien JM, Hale JF, Cahan M, Quirk M, Sullivan K, Lewis J. The impact of deliberate reflection with WISE-MD™ modules on critical thinking of nurse practitioner students: a prospective, randomized controlled pilot study. J Nurs Educ Pract. 2016;6(1):55.³⁹

Abbreviation: PE, physical examination.

Ethics

The study was exempted from review (14811) by the UMass Medical School Committee for the Protection of Human Subjects in Research.

Results

Participants

Of the 31 SC study participants, 15 (48%) participants dropped out of the study. Of the 16 (52%) remaining par ticipants, 7 participants failed to complete all parts of the study. Nine (29%) students completed the entire study – four females and five males. Participants’ mean age was 25.77 years (range =23–29 years). All nine completers had a baccalaureate degree, and one had achieved a master’s degree. Eight were Caucasian and one was of Middle Eastern descent. Of those completing, four were from the intervention group and five were from the control group.

Comparison of critical-thinking outcomes by group

The control and intervention groups showed no difference in pre- and posttest free-thought critical-thinking outcomes. SC students in the intervention group demonstrated a higher level of critical thought when prompted by questions in the DR guide.³⁹ Table 3 provides examples of SC students’ DR narratives from either the thyroid module or the appendicitis module that are representative of the critical-thinking category/subcategory.

Table 3 Critical-thinking outcomes by category and subcategory, with examples

Category	Subcategory	Selected quotations from SC students’ narrative from the two DR modules
Takes perspective	None	I think the patient is probably concerned. Any time a patient has a mass of unknown origin, there is always a concern about cancer. It would be important to educate the patient that even if this is a cancer, most types are fairly treatable. Also let her know there are many benign etiologies as well.
Considers alternatives	None	The things to be considered in the differential include possible gastroenteritis, bowel inflammatory disease, diverticulitis, possibly cholecystitis or pancreatitis and also consider intussusception and volvulus. It could also be gynecological pain, including pelvic inflammatory disease, ectopic pregnancy, ovarian cysts, endometriosis, as well as urologic symptoms; it could be a kidney stone or cystitis or pyelonephritis.
Makes associations	Describes	Again, the physician used strategies that started with open-ended questions to get the history of the patient, and then focused questions which were not discussed previously and ultimately history as well of the events that helped the physician narrow the differential down to appendicitis.
	Compares	…we will be looking at the skin, whether it is moist and warm, or cool and dry, hair course or brittle versus fine…reflexes increased could be a tip off for hyperthyroidism, and if decreased, it could be hypothyroidism
	Prioritizes/evaluates	Important lab tests to order include TSH and T3 levels: TSH is the regulatory hormone that increases or decreases thyroid function, and T3 is the amount of thyroid hormone present. If T3 is high, then TSH should be low; both of these help delineate at what level, whether it be hypothalamus, pituitary, thyroid. Given the physical exam findings included a thyroid nodule, I would think that a thyroid is a main cause of her problems.
	Integrates	For appendicitis: the history along with the fact that the patient had marked right lower quadrant tenderness to light palpation and transmission of tenderness with increased pressure. The psoas sign was positive as the thigh was extended on the right side and the obturator sign was also positive with external rotation of the leg so all of these together with the benign pelvic exam point to a diagnosis of appendicitis. The liver furthermore was not enlarged so it’s unlikely that this pain is coming from the gall bladder and the right upper quadrant was nontender to palpation. Appendicitis seems to fit best.
Anticipates outcomes	Examines assumptions	I think a lot of people hear right lower quadrant pain and automatically think of appendicitis, so it would be important to consider other possible diagnoses such as small bowel obstruction, other causes of peritonitis, ectopic pregnancy, and some type of ovarian cyst or abscess.
	Predicts outcomes	Appendicitis can be a surgical emergency that needs to be addressed relatively quickly. In terms of the pathophysiology, I am thinking about the risk of appendicitis leading to perforation and subsequent abscess formation or generalized peritonitis.
	Considers pitfalls	Given the patient is a 32-year-old woman with right lower quadrant pain, one should always be concerned for an ectopic pregnancy or some issue related to OB/GYN. These are important to consider because something like an ectopic pregnancy can be life threatening, if it’s not diagnosed correctly.
Self-assesses thinking process	Assesses self-confidence	Given the patient’s presentation and her physical exam and imaging findings, I would feel confident letting the patient know that she has appendicitis.
	Considers experience	I saw several cases of appendicitis while on my general surgery rotation and I was surprised that some of the patients were fairly comfortable when they came in with appendicitis.
	Evaluates learning style	I really like it when they have graphics and pictures of exactly where they’re showing. I feel like so much of my studying, I spend a lot of time Google imaging things and trying to find a good picture and so much time is wasted like that and I feel like it’s just so helpful when those graphics are put together and it’s just so nice to visualize everything.

Note: Adapted from Terrien JM, Hale JF, Cahan M, Quirk M, Sullivan K, Lewis J. The impact of deliberate reflection with WISE-MD™ modules on critical thinking of nurse practitioner students: a prospective, randomized controlled pilot study. J Nurs Educ Pract. 2016;6(1):55.³⁹

Abbreviations: DR, deliberate reflection; SC, surgical clerkship.

During the free-thought steps (step 1 [pretest] and step 4 [posttest]) of methods, students in both the control and intervention groups predominantly verbalized in the categories of description, learning style, and occasionally their past experiences with the module topic. They described what they were seeing and hearing in the modules as an ongoing commentary about each section of the module. They also described what they liked/did not like (in terms of their learning styles) about the module, the narrator, and the interaction between the physician and the patient. Some evaluated or summarized the overall value of the module for them at the end of their free-thought recording. The free-thought narratives did not demonstrate clinical reasoning.

SC students’ feedback on WISE modules

Because of the large dropout rate (of the 31 SC learner participants, 15 [48%] participants dropped out of the study, and of the 16 [52%] remaining participants, 7 participants failed to complete all parts of the study), the authors sought to determine 1) the reasons for the high dropout/failure to complete rate and 2) the value of the modules to all students. All participants (31) were surveyed poststudy for qualitative feedback and quantitative feedback regarding the study and the modules; 15 (48%) of the 31 participants returned the survey, but not all 15 respondents responded to every question.

Discussion

The authors were disappointed that after having been exposed twice to the DR guidance, the performance of the intervention group on the last two modules did not continue to demonstrate the same high level of critical thinking as during the DR-guided modules (steps 2 and 3). Despite the lack of support for our hypothesis, we believe that DR has value and generates the following additional questions for consideration in future studies: 1) were we measuring the right concepts? 2) are two guided DRs insufficient for students to internalize, without the DR guidance, the same high level clinical reasoning process? and 3) would a formal debriefing process (online or face to face) have enhanced the students’ experience?³⁹ Ibiapina et al,²¹ in their 2014 study of free reflection, modeled reflection, and cued reflection, found that with a higher amount of guidance, the structured reflection groups performed significantly better than the free-reflection group. These findings are consistent with our findings in that students with the guided/DR demonstrated a higher level of critical thinking versus unprompted free reflection.³⁹

We implemented the poststudy survey as an opportunity to “debrief ” the students in terms of what they did and did not find to be useful about the modules and the study. The feedback from the students in our study was similar to that found in other studies related to multimedia education enhancement strategies. Examples were as follows: 1) because of the fast pace of medical education, students become highly strategic in their selection of learning resources and unnecessary information is not appreciated;⁴⁰ it appears from the postsurvey that the SC students perceived the WISE modules to be in addition to other course expectations and students trusted their usual mode of studying and preparing for the examinations using their textbooks; and 2) the pressure of end of clerkship examination and the National Board of Medical Examiners’ (NBME) subject examination along with their perception of limited time to try out a new way of learning created barriers to their enthusiasm for the modules.⁴¹ This is consistent with Yavner et al,⁴⁰ who emphasized the importance of faculty making clear the purpose and value of any additional online initiatives. As in the Ellaway et al⁴¹ study, the students found the modules to be very useful when they had adequate time to prepare for a known specific case with which they were going to be involved. Consistent with Ellaway et al,⁴¹ SC students complained that even during “downtime” in clinical settings, they were not permitted to use electronic devices. The students expressed that this would have been an ideal time to review modules, particularly just prior to an upcoming case. Both the SC students and those of Ellaway appreciated the split screen videos of the virtual and actual surgery.⁴¹

The authors believe after completing and analyzing the data that requiring the DR guidance for every module may have increased the likelihood of students developing and internalizing a thought pattern or process that enhanced their clinical reasoning.³⁹ In addition, critical thinking as a response style may only become a habit if practiced over time. Perhaps students would need more than two opportunities for the application of guided (deliberate) reflection^24,39 suggesting that learning a new automatic pattern of thinking is enhanced by practice and observing expert modeling of critical thinking over time.^39,42

Limitations

There are limitations to this study. The pilot study was conducted at a single academic institution in New England, with a limited number of students, only 29% of whom completed the study. The time frame for the study was short, and the intervention group was instructed to use DR guidance with only two of the WISE modules. The outcomes measured were author-defined rather than consensually defined constructs: 1) critical thinking, which lacks an expert consensus definition; 2) the process of DR, developed by the study team; and 3) researcher-developed, not previously tested categories of critical thinking.³⁹ Additionally, this study was a one-time brief commitment of the entire clerkship curriculum for third-year medical students, thus, only a short-term “injection” within the four-year medical education process.²⁴

Implications for medical education

Despite the small number, high dropout rate, and lack of support for the hypothesis, we believe that this study has implications for health professions’ curricula³⁹ and provides a forum for discussion in medical education. Introducing this approach to – and level of thinking to – students ab initio might be the best way to ensure that DR remains with them throughout their academic and professional lives. Implementing the DR approach to problem solving beginning in year 1 and then threading it throughout all 4 years of medical school might enhance the impact of DR on critical thinking.

The feedback from the post-study survey was valuable in terms of a debriefing strategy. Some of the students’ feedback was similar to that found in other studies⁴¹ that participants in both the intervention and control groups valued using the WISE modules. The authors believe that the WISE modules offer SC students a unique resource for them to follow a complete interaction between an experienced surgeon and patient for the core disease processes in surgery. Students can pace themselves and use the resource 24/7. They can start and stop the program at will depending on available time to offer a chance for reflection on the covered material.³⁹ The modules are a key resource for students preparing for oral examinations as well as objective structured clinical examinations. The program is comprehensive such that all aspects of any given clinical problem are covered from epidemiology to symptoms, diagnostic workup, and treatment algorithms so that the material serves as good review for the written NBME examination as well. Many of the modules have videos and other graphic materials that can help the students review anatomy and acquire knowledge of the procedure prior to participation in the surgical suite.

Conclusion

Despite no difference in unprompted outcomes between groups, the intervention group verbalized more thoughtful clinical decision-making when following the DR protocol.³⁹ The authors now believe that limiting the application of DR with only two modules was not sufficient for students to internalize a new way of thinking about clinical cases. We suggest that DR could be integrated throughout medical education as a means to reinforce learning with a defined model to promote critical thinking for clinical reasoning. It would be of particular interest to see how many guided DR modules it might take for students to begin to verbalize and record their critical-thinking processes without prompting from the requirements of the DR protocol.³⁹

Faculty must identify and test strategies that will help learners develop and enhance good critical-thinking skills.³⁹ The value of this study is that it takes a legitimate and sponsored tool in surgical education (WISE-MD™) and poses the toughest criteria of evaluation of an educational intervention, ie, does it affect the way we think? and not just what we learn (but how we learn it)?³⁹ Finding out the answer is significant and will require more resources before we arrive at a definitive answer.³⁹

Author contributions

All authors contributed toward data analysis, drafting and revising the paper and agree to be accountable for all aspects of the work.

Acknowledgments

This work was partially supported by a $21,963.00 grant from the Institute for Innovative Technology in Medical Education (iInTIME).

The authors wish to acknowledge the SC students who participated in the study and provided important feedback and the valuable assistance and support from Victoria Rossetti, Education and Clinical Services Librarian, Lamar Soutter Library, University of Massachusetts Medical School. This article reports on the second pilot study (surgical clerkship students) that used exactly the same study design as the initial pilot with nurse practitioner students, which was published in the Journal of Nursing Education and Practice.

Disclosure

JMT served as a voluntary member of the Editorial Board of WISE-MD⁽™⁾ Leadership 2013–2017. The authors report no other conflicts of interest in this work.