Effect of a workplace-based learning program on clerkship students’ behaviors and attitudes toward evidence-based medicine practice

ABSTRACT

In clinical clerkship (CC), medical students can practice evidence-based medicine (EBM) with their assigned patients. Although CC can be a valuable opportunity for EBM education, the impact of EBM training, including long-term behavioral changes, remains unclear. One hundred and nine fourth- and fifth-year medical students undergoing CC at a medical school in Japan attended a workplace-based learning program for EBM during CC (WB-EBM), which included the practice of the five steps of EBM. The program’s effect on the students’ attitudes toward EBM in CC was assessed through questionnaires. A total of 88 medical students participated in the program. Responses to the questionnaire indicated high satisfaction with the WB-EBM program. The most common theme in students’ clinical problems with their assigned patients was the choice of treatment, followed by its effect. Based on the responses in the post-survey for the long-term effects of the program, the frequency of problem formulation and article reading tended to increase in the ‘within six months’ group comprising 18 students who participated in the WB-EBM program, compared with the control group comprising 34 students who did not. Additionally, the ability to self-assess problem formulation was significantly higher, compared with the control group. However, among 52 students who participated in the WB-EBM program more than six months later, EBM-related behavioral habits in CC and self-assessments of the five steps of EBM were not significantly different from those in the control group. The WB-EBM program was acceptable for medical students in CC. It motivated them to formulate clinical questions and enhanced their critical thinking. Moreover, the WB-EBM program can improve habits and self-evaluations about EBM. However, as its effects may not last more than six months, it may need to be repeated across departments throughout CC to change behavior in EBM practice.

KEYWORDS:

• Evidence-based medicine
• workplace-based
• clinical clerkship
• peer-review
• medical students

Introduction

In clinical practice, evidence-based medicine (EBM) is defined as medicine that uses the best available scientific information to guide appropriate clinical decisions. It is important for resolving clinical questions and promoting patient-centered care [1,2]. The practice of EBM is recommended in five steps: 1) problem formulation, 2) information searching, 3) critical examination, 4) application to patients, and 5) reflection. In recent years, the rapid increase in the amount of available medical information has been a challenge for medical education [3]. For example, erroneous medical information regarding coronavirus disease 2019 was disseminated, causing confusion even among medical professionals [4]. The practice of EBM has thus become more complex and involves lifelong learning.

Practicing EBM has become a core competency in the curricula of medical schools globally and is taught prior to and throughout clinical practice. In the UK, EBM serves as an introductory program before the start of the formal curriculum [5]. In a survey of EBM education for medical students in the US and Canada, almost all medical schools reported that EBM education was included in the regular curriculum [6]. Similar to other countries, the Model Core Curriculum for Medical Education in Japan (MCCMEJ), revised in 2016 by the Ministry of Education, Culture, Sports, Science, and Technology, included EBM education as the basis of medical treatment [7]. Although a combination of education methods, such as lectures, e-learning, group discussions, journal clubs, and the use of technology has been useful in improving EBM knowledge and skills [6], an effective teaching methodology for EBM has yet to be established [8,9]. The policies of each educational institution and their educational practices have been left to the individual educators, leading to insufficient sharing and accumulation of model practices.

In clinical clerkships (CC), medical students participate in actual medical care as student doctors in medical professionals’ teams and can raise clinical questions regarding their assigned patients. Clinically integrated EBM teaching is more effective than classroom teaching [10] and can best be taught during CC; however, it is not optimally practiced in clinical settings owing to multiple barriers, including difficulties in applying EBM skills learned in the classroom to clinical practice [11]. Apart from the setting where it is applied, the effect on students will depend on the skillfulness and eagerness of the faculty in training them [12]. In addition, education for EBM is not optimally practiced in clinical settings owing to multiple barriers, including difficulties in transferring EBM skills learned in the classroom to clinical practice [11].

In our lecture for clerkship students in respiratory medicine held in 2019 [13], we showed that lectures on literature search during CC, where medical students needed to solve clinical questions, could be effective in improving literature search literacy. However, our previous report only focused on literature search during CC, while other elements of the five steps of EBM were not adequately addressed. Considering the process of resolving clinical questions in CC, learning all five steps of EBM is encouraged [14]. Reports on EBM education being integrated into CC [15–18], have mainly focused on EBM training based on fictitious clinical scenarios. Therefore, the long-term effectiveness of EBM education needs to be verified [19].

This study developed a workplace-based learning program for EBM during CC (WB-EBM program) that integrated the five steps of EBM in CC based on the students’ assigned patients. We then evaluated the effect of our program on students’ attitudes toward EBM in CC.

Methods

Ethical approval

This study was approved by the Ethics Committee of Chiba University (approval no. 4106). The study database was anonymized. Informed consent was procured from the participants prior to their participation in the survey that was documented on the Web. All methods were performed in accordance with relevant guidelines and regulations.

Study design

This was a descriptive, interventional study comparing two groups without randomization (Figure 1). The effectiveness of the WB-EBM program was evaluated through worksheets created by the participants in the WB-EBM program and questionnaires administered to participants and non-participants. A post-questionnaire was also administered to evaluate the long-term effects of the program. The responses were compared between three groups based on the length of time since they had participated in the program, as well as with a control group that had not participated the program.

Figure 1. Flow diagram of the design.

EBM: evidence-based medicine, WB-EBM program: a workplace-based learning program for EBM during clinical clerkship.

Setting

Pre-clinical clerkship

Medical schools in Japan offer a six-year curriculum, of which two years are generally spent in CCs [15]. In Chiba University, a lecture on literature search is given in the first year as an instruction of EBM education. From the first to third year, students are assigned to laboratories to conduct research activities through a scholarship program, through which some students begin reading English medical articles. However, the educational method applied to the students is not standardized and differs depending on the instructor. Additionally, the content of the curriculum mainly focuses on basic research and does not realistically reflect clinical practice.

Clerkship

From December of the fourth year to October of the sixth year, clerkship students are rotated from one department to another every four weeks for 72 weeks in the university hospital. In the CC of the Department of Respiratory Medicine, groups of seven to eight medical students underwent a four-week training program as members of a medical team of doctors and residents. All medical students conducted daily physical examinations for two to four patients assigned to them during this four-week period. In addition to writing daily medical records, the students were required to write two case summaries within the four-week CC and include at least one reference in each summary.

Participants

Chiba University has approximately 120 medical students in each academic year. The number of participants in this study was based on realistic possibility without calculating a sample size. The sample was recruited based on 145 students who underwent CC from May 2021 to July 2022. This study included two grades of students: those who started CC in December 2020 and those who started CC in 2021. Students were able to choose CC in either the Respiratory Medicine or Thoracic Surgery departments. During the study period, 109 students underwent CC in the Department of Respiratory Medicine and attended the WB-EBM program. The remaining 36 students underwent CC for Thoracic Surgery, acting as the control group.

To evaluate the long-term effects of the program, the students who participated answered a post-questionnaire more than three months after the WB-EBM program. In addition, the control group who underwent CC in the Department of Thoracic Surgery and had not participated in the program also answered the same questionnaire.

WB-EBM program

This study evaluated the effects of the WB-EBM program on medical students in CC. The program consisted of a 60-min lecture and a 120-min group discussion delivered by one of the authors of this paper (GS, HK) during the first and third week of the CC, respectively. Table 1 summarizes the schedule and contents of the WB-EBM program.

Table 1. Time course and contents of the WB-EBM program.

1st week: First session: Lecture and practice of literature searching (60 min)

1. Overview of EBM (10 min) • Explanation of the era of evidence-free medicine through the history of scurvy treatment • Definition of EBM and the concept of evidence pyramid and its limitations • Impact Factor and its limitations 2. Introduction to making PICO/PECO (15 min) • Usefulness and limitations of secondary sources, importance of primary sources, and how to examine primary sources • Explanation of PECO/PECO 3. Introduction to literature and practice of it (35 min) • Introduction to literature search engines • Introduction to literature search methods • Practice of literature search: treatment selection using a simulated case of a patient with severe asthma

2nd and 3rd week: Self-study using worksheet

1. Each student will practice the EBM Steps based on their responsible patients. • Formation of clinical question • Literature search practices • Reading the literature 2. Summarize them on a worksheet. Formation of clinical question

Last day of 3rd week: Submission of worksheets

4th week: Second session: Group discussion with peer review (120 min)

1. Practice critical examination (30 min) • Based on online new article regarding to an original article [20] • Identification of problems as applied to PICO/PECO • Evaluation of outcomes focusing on the difference between surrogate endpoints and true endpoints • Understanding of phases in clinical research 2. Presentation of worksheets and critical examination by peer students (90 min) • Introduction (1 min) • Reading of the abstract by each participant (2 min) • Peer review and discussion of the article (2 min) • Discussion by all participants (3 min) • Commentary from the attending physician on each of the literature cited by the student (3 min)

EBM: evidence-based medicine, PICO/PECO: population, intervention/exposure, comparison, outcome, WB-EBM program, workplace-based learning program for EBM during CC.

First session

The attending doctor (GS) conducted a lecture using PowerPoint™. The first session was divided into three sections: 1. Overview of EBM (10 min); 2. Introduction to making the population, intervention/exposure, comparison, outcome (PICO/PECO) (15 min); and 3. Introduction to literature search and practice (35 min). In the first section, the definition of EBM [21] was explained by presenting the history of the treatment of scurvy and the points where progress was not made because past medicine ignored evidence. In addition, the hierarchy of evidence [22] and its limitations, and the impact factor and its limitations were explained. The second section explained the definitions of background and foreground questions, the method of formulating clinical questions following PICO/PECO framework, the usefulness and limitations of secondary sources, the importance of primary sources, and how to examine primary sources. The third section described literature search engines and literature search methods. Based on the above, a simulated case of a patient with severe asthma was presented to the students, and each student practiced searching for additional treatment options.

After the lecture, the medical students formulated clinical questions following the PICO/PECO framework based on their own cases. Based on their clinical question, they conducted a literature search, read an article, and examined the applicability of the question to their cases. They then summarized the activities on a designated worksheet (Supplementary Figure S1), which was submitted at the end of the third week of CC. The supervising doctor (HK, GS, KT) checked the worksheets and discussed them with the students during the small group meetings in the fourth week.

The worksheets in the WB-EBM program included overviews of the case, foreground question, process and methodology, details of the article (title, source, and authors), an overview of the abstract of the article, discussion by students, the degree of the question solution, and unclear points and other comments. Degree of resolution of the student’s foreground question was scored by self-assessment on a 5-point Likert scale ranging from 1 (Unresolved) to 5 (Fully resolved).

Second session

The attending doctor (GS, KT) conducted the second session, which was divided into two sections: 1. Practice critical examination (20 min), 2. Presentation of worksheets and critical examination by peer students (100 min). In the first section, the original article ‘Phase II Study on MDMA-Assisted Psychotherapy for PTSD Patients’ published in The Lancet Psychiatry [20] was introduced to the students. In the Japanese Internet news article, the news article was incorrectly described: ‘The Lancet also published a study on MDMA for PTSD patients, and MDMA will soon become a PTSD treatment drug.’ Students critically examined the article in terms of PICO/PECO, the difference between surrogate and true endpoints, and the phases according to the clinical study. They then had a group discussion on the issues in the news article. Throughout this process, students practiced identifying what was wrong with the citation, its correct interpretation, and the steps of the clinical study. In the second section, peer students discussed the articles that each student had researched, which involved the following: 1) introduction (1 min), 2) reading of the abstract by each participant (2 min), 3) peer review and discussion of the article (2 min), and 4) discussion by all participants (3 min). Up to this point, the students who had examined the paper in question moderated the discussion and examined how the paper could be interpreted by PICO/PECO. The attending doctor (GS, KT) allowed three minutes for commentary on each article.

Data collection

Evaluation of the WB-EBM program

To evaluate the effect of the WB-EBM program on the students, quantitative data were obtained from their responses to the questionnaire (Table 2): (A1) What is your level of satisfaction with the WB-EBM program? (A2) Did it motivate you to do more literature searches? (A3) Did it make you think about the correctness or incorrectness of medical information? The questions were scored on a 5-point Likert scale ranging from 1 [Extremely poor (A1); Extremely disagree (A2 and A3)] to 5 [Extremely good (A1); Extremely agree (A2 and A3)].

Table 2. Questionnaire items before and after the WB-EBM program.

After the program	Responses
(A1) What is your level of satisfaction with the WB-EBM program?	1 (extremely poor) to 5 (extremely good)
(A2) Did it motivate you to do more literature searches?	1 (extremely disagree) to 5 (extremely agree)
(A3) Did it make you think about the correctness or incorrectness of medical information?	1 (extremely disagree) to 5 (extremely agree)
Long after the program	Responses
(B1) When you have a clinical question during CC, do you formulate the question as PICO/PECO?	No, I do it when I am assigned (passively), I do it spontaneously sometimes, I do it spontaneously on a daily basis.
(B2) How often do you search the literature per month?	0, 1–2 times, 3–4 times, 5–8 times, 9 or more times
(B3) How many references do you currently read per week?	0, 1–2 times, 3–4 times, 5–8 times, 9 or more times
(B4) How often do you critically examine the medical information you have researched (not only articles)?	Never, Sometimes, Routinely.
(B5) What is your self-assessment of your competence in EBM? (B5–1) Problem formulation (B5–2) Information gathering (B5–3) Critical examination (B5–4) Application to patients (B5–5) Reflection	5-point rating scale based on the rubric in Table 3.

CC: clinical clerkship, EBM: evidence-based medicine, PICO/PECO: population, intervention/exposure, comparison, outcome, WB-EBM program, workplace-based learning program for EBM during CC.

Assessment of the worksheets

For the worksheets completed by the students through the WB-EBM program, we assessed the theme of the clinical question formulated by the students, literature search methods, type and research design of the searched literature, and year of publication. The research designs of the retrieved literature were categorized according to those generally used in the guidelines: 1) evidence from a systematic review or meta-analysis of all relevant randomized controlled trials (RCTs), 2) evidence from well-designed RCTs, 3) evidence from well-designed controlled trials without randomization, 4) evidence from well-designed case-control and cohort studies, 5) evidence from systematic reviews of descriptive and qualitative studies, 6) evidence from single descriptive or qualitative studies, and 7) evidence from the opinion of authorities and/or reports of expert committees [22,23]. Degree of resolution of the student’s foreground question was scored by self-assessment on a 5-point Likert scale ranging from 1 (Unresolved) to 5 (Fully resolved).

Evaluation of the long-term effect of the WB-EBM program

A survey was administered to all participants in November 2022. The participants included students who had attended a WB-EBM program for less than six months, six months to one year, and more than one year. The students were divided into three groups depending on the duration of their WB-EBM program: within six months, six months to one year, and more than one year.

The questionnaire included the following items regarding EBM practice in CC as shown in Table 2: (B1) When you have a clinical question during CC, do you formulate the question as PICO/PECO? (‘No,’ ‘I do it when I am assigned [passively],’ ‘I do it spontaneously sometimes,’ ‘I do it spontaneously on a daily basis’); (B2) How often do you search the literature per month? (‘0,’ ‘1–2 times,’ ‘3–4 times,’ ‘5–8 times,’ ‘9 times or more’); (B3) How many references do you currently read per week? (‘0,’ ‘1–2 times,’ ‘3–4 times,’ ‘5–8 times,’ ‘9 times or more’); (B4) How often do you critically examine the medical information you have researched (not only articles)? (‘Never,’ ‘Sometimes,’ or ‘Routinely’); and (B5) What is your self-assessment of your competency in EBM? Question (B5) was divided into the following five items: (B5–1) problem formulation, (B5–2) information gathering, (B5–3) critical examination, (B5–4) application to patients, and (B5–5) reflection. A rubric was presented for the rating, as shown in Table 3. The questionnaire items and the rubric were created by an author (GS) through the consensus of two medical education specialists (HK and SI).

Statistical analysis

The quantitative data are expressed as mean ± standard deviation (SD) unless otherwise indicated. The continuous variables were compared between the groups using the Mann – Whitney U test and the proportions of categorical variables were compared using Pearson’s chi-square test. Statistical significance was set at p < 0.05. All statistical analyses were performed using IBM SPSS for Windows, version 29.0 (IBM Company, Armonk, NY, USA).

Results

A total of 109 medical students participated in the WB-EBM program during the study period. After excluding 21 students who provided insufficient responses to the questionnaire and 11 students with missing report data, the final sample included 88 students (64 men and 24 women) for report evaluation. Figure 2 shows a flow diagram with the reasons for excluding participants from the analysis.

Figure 2. Flow diagram showing the reasons for participant exclusion from the analysis.

EBM:evidence-based medicine.

Evaluation of the education program

Seventy-nine students (89.8%) showed a high satisfaction level (scores 4 and 5) with the WB-EBM program (4.5 ± 0.7). The program highly motivated them to conduct more literature searches and consider the correctness or incorrectness of medical information (4.6 ± 0.6 and 4.6 ± 0.5, respectively).

Assessment of the worksheets

Table 4 summarizes the worksheets of the 88 students in the WB-EBM program. Regarding the theme in PICO/PECO formulated by the students, the choice (n = 27, 30.7%) and effect of treatment (n = 27, 30.7%) were the most common. Regarding the methods of literature search, keyword search by PubMed was the most common (n = 50, 56.8%), followed by requotation from guidelines on the disease of student-assigned patients (n = 24, 27.2%). Regarding the year of publication, literature published in 2011 (ten years ago from this study period) onwards accounted for 76.1% of the total (2011–2020: n = 52, 59.1%; 2021–, n = 15, 17.0%). However, a few students chose literature published before 2000 (1981–1990: n = 1, 1.1%; 1991–2000: n = 4, 4.5%). Regarding the degree of resolution of their foreground question in the self-assessment, 71.0% of the students agreed that a high degree of resolution had been achieved (5 [fully resolved]: n = 8, 10.2%; 4: n = 49, 55.7%).

Table 3. A rubric for competence in EBM.

Items	1	2	3	4	5
(B5-1) Problem formulation	Understand the concepts and contents of PICO/PECO.	←←←←←	Understands the concepts and contents of PICO/PECO and can apply them to their own problems with advice.	→→→→→	Can independently explain the concepts and contents of PICO/PECO and apply them to their own problems.
(B5-2) Information gathering	Can collect some of the information sought with advice.	←←←←←	Can collect the information sought with advice.	→→→→→	Can independently search using appropriate keywords and tools to gather the information sought.
(B5-3) Critical examination	Can partially determine the internal validity of obtained information with advice.	←←←←←	Can correctly judge the internal validity of obtained information from various perspectives with advice.	→→→→→	Can correctly judge the internal validity of obtained information from various perspectives.
(B5-4) Application to patients	Can understand that the patient’s situation and intentions influence the application of evidence.	←←←←←	Can apply evidence based on the patient’s situation and intentions with advice.	→→→→→	Can apply evidence based on the patient’s situation and intentions.
(B5–5) Reflection	Can reflect on some of the EBM steps and make partial improvements.	←←←←←	Can reflect on each EBM step and make necessary improvements.	→→→→→	Can fully reflect on each EBM step and make broad-based necessary improvements.

EBM: evidence-based medicine, PICO/PECO: population, intervention/exposure, comparison, outcome.

Evaluation of the long-term effect of the WB-EBM program

Among the 88 students, 70 students (79.5%; 56 men and 14 women) responded to a post-survey for the long-term effect of the program (WB-EBM program group). Thirty-four students who did not participate in the program owing to their assignment to CC in the Department of Thoracic Surgery (25 men and 9 women) also responded to the survey (control group). Table 5 summarizes the responses of the WB-EBM program and control groups.

Table 4. Summary of worksheets created by students through the WB-EBM program (n = 88).

Parameter	Number (%)
Primary illness of the patient in charge
Interstitial pneumonia	25 (28.7)
Pulmonary circulation disease	21 (24.1)
Lung cancer	17 (19.5)
Respiratory tract infection (other than COVID-19)	8 (9.2)
COVID-19	8 (9.2)
Chronic obstructive pulmonary disease	2 (2.3)
Others	7 (8.0)
Type of theme in PICO/PECO formulated by the students
Choice of treatments	27 (30.7)
Effects of the treatment	27 (30.7)
Side effects of the treatment	9 (10.2)
Prognosis of the disease	8 (9.1)
Complications with the disease	6 (6.8)
Diagnosis for the disease	3 (3.4)
Prevention for the disease	2 (2.2)
Risk factors for the disease	1 (1.1)
Others	3 (3.4)
Methods of literature search
Keyword search by PubMed	50 (56.8)
Requotation from guidelines on the disease of student-assigned patient	24 (27.2)
Keyword search by Google Scholar	4 (4.5)
Keyword search by general search engines	3 (3.4)
Requotation from UpToDate®	3 (3.4)
Requotation from the review article	2 (2.3)
Requotation from the textbook	1 (1.1)
Requotation from the Japanese medical website	1 (1.1)
Type and research design of the literature searched
I. Evidence from a systematic review or meta-analysis of all relevant RCTs	7 (8.0)
II. Evidence obtained from well-designed RCTs	27 (30.7)
III. Evidence obtained from well-designed controlled trials without randomization	13 (14.8)
IV. Evidence from well-designed case-control and cohort studies	33 (37.5)
V. Evidence from systematic reviews of descriptive and qualitative studies	0 (0.0)
VI. Evidence from single descriptive or qualitative studies	8 (9.1)
Publication year of the literature
1981―1990	1 (1.1)
1991―2000	4 (4.5)
2001―2010	15 (17.0)
2011―2020	52 (59.1)
2021―	15 (17.0)
Students’ self-assessment of degree of resolution of their own foreground question
5: Fully resolved	8 (10.2)
4	49 (55.7)
3	19 (21.6)
2	4 (4.4)
1: Unresolved	0 (0.0)
Blank	8 (9.1)

EBM:evidence-based medicine, PICO/PECO: population, intervention/exposure, comparison, outcome, RCTs: randomized controlled trials, WB-EBM program, workplace-based learning program for EBM during CC.

Table 5. Comparison of long-term effect of the WB-EBM program between the WB-EBM program group and control group (n = 104).

Questionnaire items	WB-EBM program group
	All (n = 70)	Less than 6 months since the program (n = 18)	6 months to 1 year since the program (n = 23)	More than 1 year since the program (n = 29)	Control group (n = 34)
(B1) When you solve to a question in CC, do you formulate the question as PICO/PECO? n (%) *
No	2 (2.9)	1 (5.6)	1 (4.4)	0 (0.0)	0 (0.0)
I do it when I am assigned (passively)	30 (42.9)	5 (27.8)	11 (47.8)	14 (48.3)	18 (52.9)
I do it spontaneously sometimes	34 (48.6)	9 (50.0)	11 (47.8)	14 (48.3)	16 (47.1)
I do it spontaneously on a daily basis	4 (5.7)	3 (16.7)	0 (0.0)	1 (3.5)	0 (0.0)
(B2) How often do you search the literature per month?
0	7 (10.0)	0 (0.0)	0(0.0)	7 (24.1)	3 (8.8)
1–2 times	46 (65.7)	13 (72.2)	17 (73.9)	16 (55.2)	20 (58.8)
3–4 times	13 (18.6)	3 (16.7)	5 (21.7)	5 (17.2)	10 (29.4)
5–8 times	2 (2.9)	2 (11.1)	0 (0.0)	0 (0.0)	1 (2.9)
9 times or more	2 (2.9)	0 (0.0)	1 (4.4)	1 (3.5)	0 (0.0)
(B3) How many references do you currently read per week? n (%)
0	33 (47.1)	7 (38.9)	8 (34.8)	18 (62.1)	15 (44.1)
1–2 times	32 (45.7)	9 (50.0)	14 (60.9)	9 (31.0)	17 (50.0)
3–4 times	4 (5.7)	1 (5.6)	1 (4.4)	2 (6.9)	2 (5.9)
5–8 times	1 (1.4)	1 (5.6)	0 (0.0)	0 (0.0)	0 (0.0)
9 times or more	0 (0.0)				0 (0.0)
(B4) How often do you critically examine the medical information you have researched (not only articles)? n (%)
Never	17 (24.3)	1 (5.6)	6 (26.1)	10 (34.5)	5 (14.7)
Sometimes	46 (65.7)	14 (77.8)	15 (65.2)	17 (58.6)	24 (70.6)
Routinely	7 (10.0)	3 (16.7)	2 (8.7)	2 (6.9)	5 (14.7)
(B5) What is your self-assessment of your competence in EBM?*
(B5-1) Problem formulation	2.8 ± 1.0	3.3 ± 0.8*	2.3 ± 1.0†	2.8 ± 0.8‡	2.5 ± 0.9
(B5-2) Information gathering	3.3 ± 0.9	3.5 ± 1.0	2.9 ± 0.9	3.4 ± 0.6‡	3.2 ± 0.9
(B5-3) Critical examination	2.7 ± 0.9	3.3 ± 0.9	2.4 ± 0.8†	2.5 ± 0.9†	2.9 ± 0.9
(B5-4) Application to patients	2.7 ± 0.9	2.8 ± 0.9	2.5 ± 1.0	2.6 ± 0.8	2.6 ± 0.9
(B5–5) Reflection	2.6 ± 0.8	2.7 ± 0.8	2.4 ± 0.9	2.7 ± 0.7	2.8 ± 0.8

*indicates a significant difference compared to control group (p < 0.05); † indicates a significant difference compared to less than 6 months since the program group (p < 0.05); ‡ indicates a significant difference compared to 6 months to 1 year since the program group (p < 0.05).

Data are presented as mean ± standard deviation.

CC: clinical clerkship, EBM: evidence-based medicine, PICO/PECO: population, intervention/exposure, comparison, outcome, WB-EBM program,workplace-based learning program for EBM during CC.

There were no significant differences in the responses of all students in the WB-EBM program and control groups regarding the formulation of the question as PICO/PECO (p = 0.330), frequency of literature search in a month (p = 0.664), number of articles read per week (p = 0.892), and the habit of critically examining the literature (p = 0.469). There was also no significant difference between the two groups in their self-evaluation of the five steps of EBM: problem formulation (WB-EBM program group, 2.8 ± 1.0 vs control group, 2.5 ± 0.9, p = 0.121), information gathering (3.3 ± 0.9 vs control group, 3.2 ± 0.9, p = 0.979), critical examination (2.7 ± 0.9 vs 2.6 ± 0.9, p = 0.255), application to patients (2.7 ± 0.9 vs 2.6 ± 0.9, p = 0.666), and reflection (2.6 ± 0.8 vs 2.8 ± 0.8, p = 0.338).

Figures 3 and 4 show a comparison of the long-term effect of the program on attitudes and behaviors of EBM practice and self-assessment of competency in EBM of the program between the WB-EBM and control groups. The frequency of problem formulation was significantly higher in the within-six months group compared with those in the control group (p = 0.026). The self-assessments of the within-six months group’s problem formulation abilities were also significantly higher than those in the control group. However, the EBM-related behavioral habits in CC of the 23 students from the six months to one year group and the 29 students from the one year later group and their self-assessments of the five steps of EBM, were not significantly different from those of the control group. The self-evaluations of the six months to one year group on problem formulation and critical examination were significantly lower than that of the within-six months group (2.3 ± 1.0 vs 3.3 ± 0.8, p = 0.001; 3.3 ± 0.9 vs 2.5 ± 0.9, p = 0.005, respectively). The self-evaluations of the six months to one year group on their problem formulation (p = 0.043) and information gathering (p = 0.037) were significantly lower than that of the more than one year group (2.3 ± 1.0 vs 2.8 ± 0.8, p = 0.043; 2.9 ± 0.9 vs 3.4 ± 0.6, p = 0.037, respectively).

Figure 3. Comparison of the long-term effect of the program for attitudes and behaviors of EBM practice between the WB-EBM and control groups (n = 104).

*indicates a significant difference compared to control group (p < 0.05). CC: clinical clerkship, EBM: evidence-based medicine, M: month, PICO/PECO: population, intervention/exposure, comparison, outcome, WB-EBM program: workplace-based learning program for EBM during CC. Y: year.

Figure 4. Comparison of the long-term effect of the program for self-assessment of competency in EBM of the program between the WB-EBM and control groups.

*indicates a significant difference compared to control group (p < 0.05); ^†indicates a significant difference compared to the less than 6 months since the program group (p<0.05); ^‡indicates a significant difference compared to 6 months to 1 year since the program group (p<0.05).

EBM: evidence-based medicine, M: month, WB-EBM program: a workplace-based learning program for EBM during CC, Y: year.

Twenty-three students (32.9%) expressed a favorable opinion of the WB-EBM program. In particular, six students (8.6%) stated that participation in the program was useful in their CC. Some of the students’ comments from the post-survey include: ‘Having learned the know-how of literature search in CC of Respiratory Medicine, I am no longer afraid of searching literature in English and can find what I want on a daily basis,’ ‘I was able to apply the mindset taught to literature search at the CC after Respiratory Medicine,’ and ‘After participating in the WB-EBM program, I was able to remember to critically examine articles when reading them during my CC in other departments.’ In addition, five students (7.1%) identified problems and areas for improvement, such as the length of class time and inadequate materials.

Discussion

This study evaluated the effectiveness of the WB-EBM program during CC. The three main findings are as follows: First, the WB-EBM program was acceptable for the medical students in CC, and it presented them with an opportunity to learn EBM in a clinical setting. In particular, the WB-EBM program can improve the ability of formulating their clinical questions. Second, medical students’ literature searches at CC were often based on requotations from secondary literature, such as guidelines. Furthermore, there was insufficient interest in the year of publication of the literature they searched. Third, while students perceived the WB-EBM program as a good opportunity to learn EBM, the long-term impact of EBM practice in CC was limited.

Although there is no established method for EBM education [8], a combination of lectures, e-learning, group discussions, journal club, and technology are considered useful for improving EBM knowledge and skills [24]. Furthermore, peer teaching has been effective in EBM education [25]. In this study, in addition to the lecture on literature search conducted in our previous report [13], the program included activities that trained medical students to formulate clinical questions, critically review the literature examined, and reflect on their assigned patients. Peer reviews were also conducted, in which students shared the literature they examined and identified areas for improvement in their critical-thinking ability. Peer review may promote metacognition through self-assessment and reflection; this would enable students to check on each other’s learning status and share information on searching for better literature that can be applied to clinical problems.

The WB-EBM program allowed students to practice formulating clinical questions, searching the literature, and determining whether the information retrieved can be applied to clinical practice. Consequently, they were able to formulate their own clinical questions related to their assigned patients and conduct EBM to resolve them, as seen in the students’ responses in the worksheets (5 [fully resolved]: n = 8 [9.2%] and 4: n = 49 [56.3%]). Some students also stated that the WB-EBM program was useful in subsequent CC.

There has been limited research on how medical students conduct literature searches during their CC. In this study, about 71.6% (63/88) of the students’ clinical questions were related to the selection, efficacy, and side effects of the treatment of their assigned patients. Before their CC, the students often used textbooks, which may contain out-of-date information, especially given the remarkable speed of treatment development in recent years [26]. Therefore, textbooks must only be used as a backbone to understanding a disease, and should always be augmented by journals and other aids to stay abreast of medical advancement [26].

While PubMed was the most commonly used search method in the present study, Gruppen et al. [27] reported that a single and brief training session could have a marked beneficial effect on the quality of subsequent and short-term EBM literature searching performance outcomes. In the report, the most common search errors were a lack of Medical Subject Headings (MeSH) explosion, missing MeSH terms, lack of appropriate limits, failure to search for the best evidence, and inappropriate combination of all search concepts [27]. Requotation from guidelines on the disease of student-assigned patients was the second most used method (30.7%; 24/88). This may be because of the availability of Japanese-language guidelines, and the fact that they contain content consistent with clinical questions about treatment selection and efficacy.

Sánchez-Mendiola et al. [18] reported that usage of the Cochrane Library and secondary journals increased after an educational intervention on EBM for CC students. Education in reading English language articles can be a barrier for EBM education in non-English speaking countries [11]. Therefore, guidelines in Japanese have been an important means of accessing the information sought by students. In fact, the Japanese-language guidelines for lung cancer and pulmonary hypertension, which were frequently cited by the students in this study, were available free of charge, and may have been an accessible source of information. While guidelines can certainly be useful, their contents can quickly become outdated [28,29]. In fact, 22.7% (20/88) of the references examined by the students were older than 2010. Therefore, instructing students to use more recent references as much as possible is crucial. In addition, experience in literature searches without using guidelines is also necessary because there are many relatively rare diseases for which guidelines do not exist.

More than six months after the WB-EBM program, there was no significant difference in the behavior of students who did not participate in the program, as well as in their self-evaluation of EBM, such as literature search and number of articles read. Furthermore, their self-evaluations of problem formulation, literature search, and critical examination tended to decrease over time. Since the WB-EBM program was conducted sequentially during the 72-week CC in the first year of the program, it was only done in one department. Thereafter, it was up to the students whether they would continue their EBM practice. Therefore, while the WB-EBM program could lead to long-term behavioral changes in EBM practice, its effects may only show limited continuity. The previous report found that medical students’ literature search literacy may not improve after a single educational opportunity [15]. Furthermore, EBM education needs to move away from an isolated and/or transient approach [30], and toward a more longitudinal one throughout medical education [12]. Adopting a scaffolded curriculum integration approach during the preclinical years can increase – or at least maintain – students’ ability to apply EBM concepts to patient care [30]. Further improvements are needed to improve long-term effectiveness.

A curriculum that provides students with opportunities to practice EBM throughout the CC period should be considered. A student who had attended the WB-EBM program commented in the post-survey, ‘I learned a lot, but I thought it would be better if there were similar workshops in other departments so that we could better practice EBM according to our respective departments.’ Another student commented, ‘I felt that the lecture during the CC was more useful than the lecture given in the classroom before the CC, because I could understand the lecture more deeply.’ In the future, students will practice the lecture component of the WB-EBM program immediately prior to the CC period. Following this, each department will repeat group discussions on the formulation of clinical problems, literature searches, and critical reviews of the literature in accordance with the problems in their respective fields. This continuous approach may influence medical students’ EBM practice. Additionally, tracking the changes in learning effectiveness for the EBM habits of the same population over time should also be considered.

The present study has some limitations. First, as it was carried out at just one medical school in Japan, its effects on participants and their comments are subject to cultural bias. Second, regarding research design, this single-site study only sampled a few participants in an uncontrolled environment and relied partly on students’ self-assessments for data collection. Third, the study was only conducted in a single department. The questionnaire was administered when medical students doing CC were rotated to the Department of Respiratory Medicine. Therefore, what they learned from the previous departments may have influenced their responses. Fourth, the questionnaire items and the rubric in this study were not evaluated for validity and reliability. In the future, the WB-EBM program and assessment of EBM competencies must be evaluated using a validated consensus-based test, such as the Fresno Test [31] and the Berlin Questionnaire [32]. Fifth, the time taken for the self-study of EBM before and during CC can differ for each student; however, this was not assessed in this study. Sixth, because this study followed a cross-sectional design, the long-term effects were compared between groups comprising different participants. In addition, the content of the CC of Thoracic Surgery, in which the students in the control group rotated, was different from that of Respiratory Medicine. Although EBM-focused education was not included in Respiratory Medicine and Thoracic Surgery other than the WB-EBM program, individual supervisors may have provided instructions on EBM. However, the influence of such independent instructions on the results was not excluded.

Conclusions

The WB-EBM program during CC motivated medical students to formulate and critically examine their clinical questions. Although medical students used requotation from secondary literature, such as guidelines and PubMed searches, they had limited interest in the year of publication of the literature. Furthermore, the long-term effect of the WB-EBM programs on students’ behavior and self-evaluation regarding EBM practice can last for several months, but such effects may diminish or disappear afterwards. Therefore, WB-EBM programs may need to be repeated across departments over a CC period to effectively change behaviors in the practice of EBM.

Acknowledgments

We would also like to thank Editage (www.editage.com) for English language editing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/10872981.2024.2357411

Additional information

Funding

This work was supported by JSPS KAKENHI Grant number JP21K17244.