Assessing trainee critical thinking skills using a novel interactive online learning tool

ABSTRACT

Background

Critical thinking is essential for the accurate diagnosis and management of patients. It is correlated with academic success.

Objective

Our objective was to design a novel tool for interactive online learning to improve knowledge and to assess trainees’ critical thinking skills using the framework of the American Philosophical Association (APA).

Methods

Residents, fellows and students participated in an online, self-directed case-based vignette activity to learn malaria diagnosis and management. Pre and post-tests with multiple choice and open-ended case-based questions assessed knowledge and critical thinking. Comparison between pre and post-test scores across subgroups were performed using paired t-tests or one-way ANOVA.

Results

Between 4 April 2017 to 14 July 2019, 62 of 75 (82%) eligible subjects completed both the pre and the post-test. Improved post-test scores occurred in 90% of medical students, p=0.001, 77% of residents, p<0.001, 60% of fellows, p=0.72 and 75% of trainees overall, p=<0.001. Fellows had higher pre-test scores than students or residents but there was no difference by level of training on the post-test.

Conclusions

This interactive online learning activity effectively imparted medical knowledge and improved trainee responses to questions requiring critical thinking. To our knowledge, this is the first time the APA’s critical thinking framework has been incorporated into interactive online learning and assessment of critical thinking skills in medical trainees. We applied this innovation specifically in global health education, but there is obvious potential to expand it to a wide variety of areas of clinical training.

KEYWORDS:

• Critical thinking
• online learning
• medical education
• malaria
• assessment

Introduction

In addition to medical knowledge, critical thinking is essential for the accurate diagnosis and management of patients [1,2]. Critical thinking is purposeful, reflective judgment based on the consideration of evidence, methods and standards in decision making [3]. Six core skills are the foundation of critical thinking: interpretation, analysis, evaluation, inference, explanation, and self-regulation [4]. This is the conceptual framework for this study.

Pediatricians must be able to care for immigrant children and returning pediatric travelers with global healthcare issues [5–7]. The Federation of Pediatric Organizations recommends that all pediatric residents be provided with training in global child health [8]. However, in 2016, 36.5% of pediatric residents performing international electives did not receive pre-travel preparation, constituting a gap in global health education [9]. As such, we sought to develop and implement a web-based interactive malaria vignette player (MVP) to provide medical knowledge about malaria and to assess the critical thinking skills of medical students, residents, and fellows.

Education about malaria is a key pillar in any global health curriculum. Malaria is a potentially fatal infection which is endemic throughout the world and infects 200 million people every year. Most malaria deaths worldwide occur in children under 5 years of age [10]. In 2018 there were 1,823 malaria cases reported to The Centers for Disease Control and Prevention (CDC) including 294 in children, 48 of whom had severe malaria. Malaria patients require timely evaluation and treatment [11]. The diagnosis and management of malaria can be challenging, particularly for new health care providers, including medical students, residents, and fellows and requires both adequate medical knowledge and critical thinking skills.

We planned to use the asynchronous delivery of the MVP with immediate online feedback tailored to the learner, to provide individualized and adaptable delivery of global health curricula to the learners [12]. Critical thinking is challenging to teach and to assess [1,13,14]. We sought to incorporate the following strategies in the (MVP) which are important in teaching critical thinking: case-based learning, asynchronous delivery (which allows the learner to tailor the pace of learning to maximize the incorporation and application of knowledge) and virtual simulation in problem solving, which enhances the active engagement of the learner [1]. To the best of our knowledge, this is the first time that the American Philosophical Association’s critical thinking framework is being adapted for the assessment of critical thinking in medical trainees via interactive online learning.

Methods

We created an online individualized learning tool, consisting of a case-based branching scenario for teaching malaria recognition, evaluation, and management [15–17]. In this scenario, each trainee was allowed to ‘choose their own adventure’ and select the region of the world in which they wished to work. Trainees were given immediate online feedback following their selections of diagnostic, therapeutic and disposition choices customized for the management of their online animated patients. We incorporated two custom animated avatars to keep the trainees engaged during the virtual simulation.

The study protocol was submitted to the Institutional Review Board of Children’s National Hospital and was determined to be exempt. All 62 individuals agreed to participate in the study. All subjects completed a Demographic Questionnaire and a pre-test prior to accessing the MVP; immediately after completing the MVP, all subjects completed a post-test, similar in content with the pre-test.

No identifying information was collected about the participants and participation was voluntary. No IP addresses were collected, and each participant was assigned a study-specific ID number to identify their pre- and post-test answers.

All attendees of the Children’s National Global Health Course, which is held semi-annually, were invited to access the module during a Global Health Course session. Trainees preparing for an international elective or those interested in increasing their knowledge about malaria were able to access the MVP, through our internal Children’s National Hospital’s online learning platform, without having an affiliation to the Global Health Course.

The pre and post- test were designed to assess residents’, fellows’ and students’ critical thinking skills as defined by the APA. The items in the pre- and post-MVP test included open-ended questions regarding the management of patients in clinical scenarios. Interpretation was evaluated by testing the trainees’ ability to understand and express the meaning or significance of situations, data, experiences or events [18] by answering questions such as: ‘What’s happening?’ ‘What does this mean?’ [19] Analysis was evaluated by testing the trainees’ ability to identify the relationships among statements, questions and descriptions [18] by answering questions such as: ‘What is your basis for saying that?’ ‘What assumptions must you make to accept that conclusion?’ [19] Explanation was evaluated by testing the trainees’ ability to list the criteria for their conclusions and to present arguments supporting those conclusions [18] by answering the question: ‘Why do you think that was the right answer?’ [19] The Pre and Post-Test Question Subject and Content is shown in Table 1.

Table 1. Pre and post-test question subject and content.

Subject of Question	Content of Question
Medical Knowledge
	Epidemiology of P. falciparum malaria
	Clinical presentation of severe malaria
	Treatment of malaria
Critical Thinking Skill Explanation	Definition: Provide a perspective of the ‘big picture’ and specify the criteria for one’s conclusions
	Provide rationale (criteria) for selecting a drug to treat severe malaria
	Provide recommendations for preparing a family for travel to a malaria endemic region and provide reasoning behind recommendations
Critical Thinking Skill Interpretation	Definition: Understand what is happening in a situation and clarify its meaning or significance
	Clarify what is happening in a clinical scenario
	Provide a differential diagnosis for the presentation in a clinical scenario including the reasoning for considering various diagnoses
	Understand what is happening to a patient and order appropriate tests to confirm the patient’s diagnosis
Critical Thinking Skill Analysis	Definition: Identify the relationship among statements or descriptions and provide arguments for coming to a given conclusion
	Identify the relationship between the administration of an anti-malarial drug and changes in a patient’s EKG
	Identify what assumptions are being made to reach the conclusion about the relationship of the anti-malarial drug and the changes in the EKG
	Identify how to test the accuracy of the assumptions that are being made to reach the conclusion about the relationship of the anti-malarial drug and the changes in the EKG

Validity

Critical thinking questions were formulated according to the Test Manual for The California Critical Thinking Skills Test (CCTST) [19], a validated tool to measure critical thinking [20], which has been correlated with academic success among medical trainees [20,21].

The pre and post-test were validated via comparison of the test results among multiple groups, i.e., medical students, residents and fellows. The test questions covered the content of the module, and their subject matter was reviewed by five malaria experts in relation to the objectives of the module.

Grading of tests

A grading rubric contained the answers offered by the five malaria experts to the medical knowledge questions (multiple choice with single correct answer) and the critical thinking questions (open-ended essay with multiple items listed by the experts in their answer to each question). For example, the experts listed several diagnostic tests to be performed in the evaluation of a patient with severe malaria. The collective answers of the experts for each question were noted and the trainee was awarded 1 point for each of their responses matching an expert’s response to that question.

Grading of the multiple medical knowledge questions was based on comparison to infectious disease references [15] and Centers for Disease Control and Prevention recommendations [16,17]. Grading of trainees’ answers to critical thinking questions was done by comparison of the trainee’s response to the compiled answers of five malaria experts. The trainee was given one point for each response which matched the responses of the experts. Grading was performed by three of the authors: an infectious disease fellow (YH), a hospitalist attending physician (PB), and an infectious disease attending physician (BJ).

Reliability assessment

Interrater reliability was assessed for our exam total scores and for each question and demonstrated high rater agreement. The median and IQR for the pretests total score were: 16 (13–19), 15 (12–17), 17 (14–20). Pairwise correlations and pairwise Intra-class correlation coefficients (ICC) ranged from 0.96 to 0.98 and the overall ICC across the three raters was 0.92. The ICCs for each pretest question that required a score ranged from 0.75 to 0.96. The median and IQR for the posttests total score were: 19 (16–23), 18 (15–21), 19.5 (17–24). Pairwise correlations ranged from 0.92 to 0.94 and pairwise ICCs ranged from 0.92 to 0.93 and the overall ICC across the three raters was 0.88. The ICCs for each posttest question that required a score were all above 0.7 except for Question 4 where it was 0.58. We also compared the change from pre to post for the total score with the following results: 4 (0–6), 3.5 (0–6), 3 (1–6). Pairwise correlations ranged from 0.88 to 0.90 and pairwise ICCs ranged from 0.88 to 0.89 and the overall ICC across the three raters was 0.89. Given the excellent reliability between raters, further analysis was done using the scoring of the infectious disease attending physician.

Comparisons between pre and post-test results were performed using paired t-tests. Comparisons between pre- and post-test across items and total scores across subgroups were performed using t-tests or one-way ANOVA. Analysis of binary items was conducted using chi-squared tests.

Results

Between 4 April 2017 and 14 July 2019, of the 76 trainees eligible to participate, 62 trainees (11 medical students, 36 residents, and 15 subspecialty fellows), 62/76 (82%) completed all study procedures. The majority 33/62 (53%) completed the MVP through Children’s National semi-annual Global Health Course, 24/62 (39%) accessed the MVP independently to improve their knowledge of global health and 5/62 (8%) completed the MVP in preparation for an international elective. Most trainees in the group 40/62 (64%) had previously cared for a patient with malaria including: 86% (13/15) of fellows, 58% (21/36) of residents and 54% (6/11) of medical students. In addition, 19/62 (31%) had worked in a malarious region.

Results are presented as means and standard deviations or medians and interquartile ranges. The improvement in trainees’ scores is shown in Table 2. Overall, the mean score for the pretest was 16.8 with the post-test improving to 20.0 which was significant at p < 0.001. Specific questions that exhibited the most improvement were: Q2 (p = 0.002) and Q7 (p < 0.001) where subjects were asked for the best treatment option for a given scenario, Q6 (p < 0.001) which asked what diagnostic testing was appropriate for a given scenario; and Q3 (p < 0.001) where the subject was asked to justify their treatment choice.

Table 2. Pre and post test and change comparisons in trainees.

Outcome	Pre	Post	Difference	p-value
	Mean (95% CI)	Mean (95% CI)	Mean (95% CI)
Medical Students (n = 11)	15.3 (12.2–18.4)	20.3 (16.7–23.9)	5.0 (2.5–7.50)	0.001
Residents (n = 36)	15.6 (14.3–17.0)	19.3 (17.8–20.8)	3.67 (2.2–5.1) 33.66	<0.001
Fellows (n = 15)	20.8 (18.9–22.7)	21.3 (18.5–24.1)	0.47 (−2.25–3.18)	0.72
Total (n = 62)	16.8 (15.6–18.0)	20.0 (18.7–21.2)	3.1 (2.0–4.3)	<0.001

Fellows had higher pre-test scores overall, p < 0.001, when compared to medical students, p = 0.007 or residents, p = 0.001. This difference was no longer reflected in the post-test scores. Trainees who had taken care of a patient with malaria (n = 40), had higher scores on the pre-test, than those who had not, p = 0.007. This difference was no longer reflected in the post-test scores. The MVP elevated the performance of medical students and residents to that of fellows. It enhanced the ability of those who had never cared for a patient with malaria to perform as well as those who had.

Trainees commented that they: ‘ … liked the interactive nature of the vignettes and the immediate feedback on … answer choices.’ They enjoyed the autonomy, especially the ‘choose your adventure aspect.’ They ‘…loved the avatars!’

Discussion

We adapted the APA critical thinking framework, which is new to the online medical education literature, to assess the critical thinking skills of undergraduate and graduate medical trainees in a novel way via interactive online learning to address a global health priority. The MVP effectively imparted medical knowledge and improved responses to questions requiring the use of critical thinking skills for care of pediatric patients with malaria. Upon completing the MVP, the performance of medical students and residents was elevated to that of fellows. The MVP also enhanced the ability of those who had never cared for a patient with malaria to perform as well as those who had.

Medical students showed the greatest improvement with 90% of them having improved post-test scores. This study demonstrates how medical students can benefit from interactive case-based training. Medical students like the interactive aspect of e-learning and the convenience and flexibility that it provides [22]. The attributes of asynchronous and interactive e-learning contribute to successful educational experiences for medical students [23]. In our study, these attributes were also appreciated by trainees at the resident and fellow level. Almost all fellows (86%) who viewed the MVP had previous experience caring for a patient with malaria and had high scores on the pre-test, as compared to the medical student and resident trainee groups. This may account for the relatively decreased amount of improvement they demonstrated on the post-test, as compared to the medical student and resident trainee groups.

Limitations of the study are that the number of participants is small and the number in each trainee group is even smaller.

Online learning is an effective modality to impart knowledge to medical trainees [24]. It is a timely and appropriate modality to teach and to assess critical thinking in medical trainees today. Online learning enables educators to provide self-directed learning, flexibility, and accessibility for medical trainees [25]. It enables trainees to control the pace of their learning and thus to customize their learning experience to fulfill their individualized goals [26].

Critical thinking is essential for the accurate diagnosis and management of patients [1,2] but it has not been fully embodied in medical education [2,27]. It has been incorporated into nursing [28,29] and allied health education [30,31]. As Huang et al [1] emphasize, the incorporation of critical thinking into the medical curriculum is of paramount importance. Critical thinking is necessary for prudent diagnostic stewardship [32], yet it is difficult to teach and to assess [1,13,14]. However, we designed innovative online learning to teach and to assess critical thinking skills, which was embraced by trainees. By incorporating basic strategies identified by Huang [1] and Chacon [27] as being important to teaching critical thinking, we devised unique means to engage and immerse the learner in the clinical milieu through virtual simulation, avatars, and immediate online formative feedback.

Incorporation of the APA’s critical thinking framework into interactive online learning as exemplified in this study and described at the beginning of the Discussion, provides a template for the assessment of critical thinking in undergraduate and graduate medical trainees. The study findings suggest that when faculty are using an online module to assess critical thinking they should consider the following: 1) develop an asynchronous web-based tool to provide individualized learning and flexibility with available access 24/7, 2) enable self-directed learning with a ‘choose your adventure’ scenario with branching pathways allowing autonomy for the learner 3) allow the learner to navigate the module at his or her own pace 4) engage the learner by a unique communication technique (for example we used avatars) 5) immerse the learner in clinical scenarios by allowing the learner to be virtually present in the clinical arena (such as having the trainee visit the emergency room, laboratory and pharmacy in the MVP) 6) provide formative feedback by having the learner select a diagnostic test, treatment option or disposition for a patient and then receive immediate interactive online feedback. In devising a pre-test or post-test to assess critical thinking, one should provide clinical scenarios with specific questions tailored to assess the selected core critical thinking skills that are unique to their specialty, that one wishes to study. For example, we focused on interpretation, analysis, and explanation, as they are of particular importance in the practice of infectious diseases. One should use the basis of the CCTST [19] questioning technique with specific generic prompters for the assessment of specific critical thinking skills, as described earlier in the Methods section.

In conclusion, to our knowledge, this is the first time the APA’s critical thinking framework was incorporated into interactive online learning and assessment of critical thinking skills in medical trainees. We applied this innovation specifically in global health education, but there is obvious potential to expand it to a wide variety of areas of clinical training. Future studies should examine the role of incorporation of the APA’s six critical thinking skills into core curricula in selected clinical rotations, with applications to diagnostic stewardship and the practice of high value care while determining a definitive diagnosis or management plan.

Acknowledgments

The authors would like to acknowledge Drs. Elizabeth Barnett, Kathy Ferrer, Andrea Hahn, Rebecca Levorson, and Lineo Thahane for being the malaria experts who reviewed and took the test and upon whose response trainee answers were graded. The authors would like to thank Mr. Jeff Sestokas who designed and built the Malaria Vignette Player.

Disclosure statement

Dr. Priti Bhansali, as the Director of the Children’s Academy of Pediatric Educators (CAPE), participated in the decision-making process to award funding for statistical analysis for this project.

Data availability statement

The data tables are stored on our secured CNH servers. However, it should be noted that we did not include obtaining permission to share the data outside of the institution on our consent forms.

Additional information

Funding

All phases of this study were supported by a Children’s National Hospital Washington, DC, Children’s Academy of Pediatric Educators (CAPE) Project Fund Grant.