A descriptive analysis of a novel intervention to help residents become evidence users

Abstract

Background and objectives: To evaluate the educational and clinical effectiveness of the ‘Brief Evidence-Based Assessment of Research’ (BEAR), a template to assist residents in searching, evaluating and integrating relevant medical literature into daily practice.

Methods: We completed a descriptive analysis of BEARs submitted by first year residents between 2005 and 2007 at the University of Alberta Family Medicine Residency program.

Results: 317 BEARs were analyzed. The most common type of question for which information was searched was therapy (59%). Residents searched Pubmed most often (38%) followed by Summary (i.e. Clinical Evidence) (22%) and Filtered sites (i.e. ACP Journal Club) (19%). Original research articles were the largest resource category used to answer questions (41%). Secondary peer-reviewed resources (filtered articles, summary sites, reviews/meta-analysis and guidelines) accounted for 48% of all resources used. 19% of residents reported a large change in practice with completion of the BEAR, 50% reported a small change, 12% stated they were reassured and 8% reported that the intervention was of no help to them.

Conclusions: The BEAR facilitates the use of a variety of resources in answering clinical questions. 69% of users reported at least a small change in clinical practice, suggesting that the BEAR may be a useful tool in evidence-based resident education.

Introduction

Despite the development of numerous evidence-based resources for medical learners and physicians, gaps persist between evidence-based medical practice and care delivered (Knapp 2008). Traditional teaching has emphasized critical appraisal as a means of answering clinical questions. However, teaching critical appraisal alone has failed to demonstrated a difference in behaviour and only small changes in knowledge (Norman 1998; Green 1999)

Some have advocated that instead of focusing primarily on critical appraisal, information management should be an integral part of teaching students how to keep up to date with recent medical advances, specifically using summarized and filtered resources (White 2004; Slawson 2005).

Translating these findings to graduate medical education suggests a de-emphasis of critical appraisal, with an increased focus on point-of-care evidence-based practice. A clinically integrated evidence-based medicine (EBM) curriculum is more likely than standalone teaching to bring about changes in desired attitudes, skills and behaviours (Coomarasamy 2004) and incorporation of clinical question exercises into medical curricula has been recommended as one method to encourage real-time evidence-based practice (Whitcomb 2005; Hatala 2006).

In response to the above issues, the Department of Family Medicine at the University of Alberta developed an integrated EBM curriculum. The intent was to facilitate efficient literature searches, increase familiarity with summarized evidence-based resources and enhance skills of rapidly synthesizing evidence from a variety of sources. In effect, we encouraged residents to become evidence users.

Table

Practice points

The BEAR may be a useful educational tool to assist residents in becoming familiar with a variety of electronic medical resources. The utilization of an educational tool to search and evaluate the medical literature results in a modest change in resident's current practice.

The objective of this study is to report on the results from the first two years of the Brief Evidence-based Assessment of Research (BEAR), including the type of questions, the resources used and the utility of project.

Methods

The BEAR is one component of a new integrated EBM curriculum that was introduced to the Department of Family Medicine at the University of Alberta in July 2005. The components of the integrated curriculum have been previously published (Allan 2008).

First year Family Medicine residents are encouraged to complete four BEARs during their 16 week family medicine rotation (one/month). Residents are provided with sample BEAR projects, a guide to completing the BEAR and formal instruction at an EBM workshop early in their training. Residents have access to the University of Alberta library website which provides access to a large volume of websites and journals. Residents were encouraged to utilize a variety of resources (Table 1). Use of secondary peer-reviewed resources was promoted during the EBM workshop. In addition, all residents received a tutorial from a librarian during the workshop, highlighting peer-reviewed evidence resources such as ACP Journal Club, Evidence Based Medicine and DynaMed. Finally, secondary peer-reviewed resources were incorporated into the BEAR template to remind residents of available options (Figure 1). Residents were encouraged to complete the BEAR at the point of care, although the time to complete each BEAR was not officially recorded. Emphasis was placed on gaining familiarity with secondary peer-reviewed resources in order to access information in a timely manner. Residents were asked to present their findings at one of five academic training sites or equivalent community site during rounds. Rounds generally consisted of a review of the completed BEAR in an informal setting with faculty and community physicians. A faculty development seminar on both the BEAR and evidence-based resources was offered to all preceptors on two separate occasions. BEARs were assessed by the program primarily on whether they were completed or not.

Figure 1. Bear work-sheet.

Table 1  Classification of information resources for the purpose of the brief evidence-based assessment of research

Resource	Description (examples)	Secondary peer-reviewed resource
Abstract	Brief summary of a research article, thesis, review or conference proceeding – usually included when full article unavailable to resident at time of search	X
Paper	Original research article such as those found on PubMed or equivalent	X
Filtered Article	Articles on specific studies that have been filtered for quality and applicability (ACP Journal Club, Evidence Based Medicine Journal)	√
Summary	Web-based resources providing evidence-focused summaries of medical topics. Often referred to as online medical textbooks that are frequently updated. (Up-To-Date, DynaMed)	√
Review/Meta-analysis	Systematic reviews on specific clinical questions (Cochrane)	√
College/Society/Guideline Paper	Systematically developed statements to assist practitioners in the provision of appropriate health care (The Society of Obstetricians and Gynaecologists of Canada Guidelines)	√

All completed BEARs for 2005–06 and 2006–07 were collected by the academic administrator at each teaching site following their presentation at rounds, and forwarded to the central family medicine residency office. The total number of submitted BEARs over a two-year period was analyzed. All data from each BEAR was entered into an excel spreadsheet and multiple variables were extracted. Resident classification of resources was primarily used unless there was a clear discrepancy between the resource cited and the actual citation listed. For instance, a resident may have reported a resource as a “paper” however if the citation indicated it was a systematic review, the resource would be classified as a systematic review in our database. The types of questions asked were categorized within critical appraisal categories (e.g. therapy, diagnosis, prognosis) and specific topic areas (e.g. cardiology, infectious disease, obstetrics/gynaecology).

The number and type of resources used were recorded. We considered filtered articles, summary sites, systematic review/meta-analysis and guidelines all to be secondary peer-reviewed resources. Results were analyzed using SPSS. Pearson correlation was used to assess for significance regarding searches used and change to practice. Ethics was obtained from the Health Research Ethics Board at the University of Alberta.

Results

In 2005 and 2006 there were 64 and 65 residents, respectively, in first year Family Medicine at the University of Alberta. The majority of residents were in the urban program, located at one of five academic sites in Edmonton. Twenty-four were rural residents situated throughout Alberta (12 in both 2005 and 2006). BEARs were available from 78% of the residents (100/129) of which 58 were female and 42 male. A total of 317 BEARs were reviewed, with a mean of 3 BEARs per resident (range 1–8). The rural residents completed a mean of 5 BEARs per resident (range 2 to 8). Although most residents submitted at least one BEAR, they were less likely to submit each subsequent BEAR assignment with only 38% submitting the ‘recommended’ fourth BEAR.

Residents reported searching Pubmed most often (38%) followed by Summary sites (22%) and Filtered articles (19%). Information on the number of resources reviewed before the top three were recorded was available from 71% of the BEARs, with a mean of 4.3 resources per BEAR (SD = 2.2–5.4; range = 1–10). Table 2 shows the types of resources residents used to complete the BEAR.

Table 2  Resource categories used by first-year family medicine residents during completion of the BEAR

Resource	Total number (%)
Abstract only*	76 (9)
Original Research*	350 (41)
Filtered Article*†	80 (9)
Summary Site*†	62 (7)
Systematic Review*†	204 (24)
College/Guidelines*†	61 (7)
Other	14 (2)
Total	847

*See Table 1 for definitions.

†Secondary peer-reviewed resources.

Residents used 285 different specific resources 760 times to answer 317 questions. Table 3 shows the top twenty specific resources reported by the residents. The top 20 resources were accessed frequently, accounting for 46% of the resources used. The remaining 265 resources not listed primarily include a broad range of peer-reviewed journals (accessed via the internet). A number of the journals listed were not specifically directed towards family practitioners (examples include: Veterinary & Human Toxicology, Journal of the Renin-Angiotensinogen-Aldosterone System, Journal of Bone and Mineral Research, Helicobacter). One reference listed a website that was an advertisement for a weight loss program. Three residents listed a search engine (i.e. Google) as their resource.

Table 3  Top 20 specific resources used by first-year family medicine residents to answer clinical questions

Resource	Total number of times used (%)
Cochrane	68 (9)
Up To Date	37 (5)
Journal of the American Medical Association	31 (4)
New England Journal of Medicine	30 (4)
Lancet	23 (3)
British Medical Journal	21 (3)
Canadian Medical Association Journal	18 (2)
American College of Physicians Journal Club	14 (2)
Clinical Evidence	12 (2)
Archives of Internal Medicine	11 (1)
Curbside Consults	11 (1)
American Family Physician	10 (1)
American Journal of Medicine	10 (1)
Journal of the American Academy of Dermatology	8 (1)
Cognitive Behavioral Therapy	8 (1)
Obstetrics & Gynecology	8 (1)
Neurology	8 (1)
Annals of Intern Med	7 (1)
Chest	7 (1)
Pediatrics	7 (1)
All others	411 (54)
Total	760 (100)

Original research articles represent the most common resource category used (41%). Many of the individual secondary peer-reviewed resources were used minimally (e.g. Evidence-Based Medicine Journal was visited only four times). However, in its entirety, secondary peer-reviewed resources accounted for almost half (48%) of the total resources used.

A paired samples t-test was completed and found there was no difference in sites searched from the first BEAR to the fourth BEAR. In light of the low number of residents who completed the fourth BEAR, the first BEAR was also compared to the third BEAR and also found to have no statistically significant difference in sites searched. The types of resources used also did not vary from first to last BEAR. For example, original articles were used in 49% of the first, 47% of the third and 48% of the fourth BEAR.

Categorized by specialty, the most common question topics were Cardiology (15%) and Obstetrics/Gynecology (11%). Using critical appraisal categories, by far the most common type of question were therapy (59%). Generally, questions were quite varied over multiple areas and reflected the broad complexity of primary care. Examples of questions are provided in Box 1.

Box 1 Sample clinical questions addressed in the BEARs

What is the prognosis of patients with white coat hypertension?

What is the best treatment for herpetic neuralgia?

Does exercise really improve outcomes in fibromyalgia?

Are oral antihistamines effective in the management of eczema?

How often does TSH need to be monitored in a patient on synthroid?

Does voiding after intercourse help prevent UTIs in women?

In hypertensive women of childbearing age what is the absolute risk increase while taking exogenous estrogens?

Does suctioning of the oropharynx and nasopharynx of a meconium stained infant at the perineum reduce infant mortality?

Change in practice was recorded on the bottom of the BEAR template after residents had completed the BEAR (Figure 1). 19% of residents reported a large change in practice after completing one BEAR, 50% reported a small change, 24% said they were reassured and 8% reported no help. There was no significant correlation between the resources searched (e.g. Pubmed versus summary sites) and change to practice.

Discussion

Residents used 285 different resources to answer 317 questions while using the BEAR. Original research articles were most commonly used, although secondary peer-reviewed resources all together were used almost 50% of the time. Residents’ continued dependence on original research articles is likely multi-faceted. Encouraging the use of secondary peer-reviewed resources has not traditionally been an educational objective, although an EBM workshop earlier in our residency program was developed in part to promote accessible EBM resources and encourage their use. This single introduction alone is likely insufficient to result in adoption. Further steps are required to encourage resident use of secondary peer-reviewed resources. These include promotion of reliable resources at the EBM workshop, incorporation of peer-reviewed resources into monthly journal clubs and academic half days as well as increasing availability of these resources to the residents primarily by electronic means. In addition, encouraging faculty to promote the use of secondary resources when giving feedback to residents about their BEARs. It is interesting that although residents were encouraged to use up to three different resources, the majority of them reported viewing a mean of 4.3 resources per BEAR. The exact reason for this is uncertain but may reflect residents’ uncertainty about which sites to trust and hesitancy to trust another's assessment of the literature, or difficulty locating an appropriate answer.

The reason for higher return of BEARs from rural residents compared to urban residents is unclear. At the time of data collection the rural program director was a strong advocate for the BEARs and encouraged their use beyond academic requirements. This may have impacted rates of completion in the rural program. Almost 70% of the BEAR projects resulted in a self-assessed change in practice and only 8% were no help, indicating the residents themselves usually found clinical utility in the exercise.

It is important to note the limitations of our study. Unfortunately, only 78% of residents formally submitted at least one BEAR worksheet and only 38% completed a total of 4 BEARs. We did not record the time residents took to complete the BEAR, which would be important in helping to assess the sustainability of this approach in clinical practice. In addition, while faculty at each academic site reviewed the BEARs with each learner, we did not centrally review the quality of the answers reported. Thus, we cannot assess if residents were reporting changes to their practice based on low or high-quality evidence. We hope to address this issue in future studies with focus on the quality of evidence with which residents are making their decisions. Finally, we did not specifically ask residents to report resources in the order they were accessed. Thus, it is possible that residents actually utilized secondary peer-reviewed literature first and then identified original papers from that secondary resource to include on the BEAR. This would in fact under-represent the utilization of secondary resources.

Conclusion

The broad use of resources including secondary peer-reviewed resources seen with the BEAR suggest that the BEAR may be a useful educational tool to assist residents in utilizing and becoming familiar with a variety of electronic medical information resources. This training during residency might help eliminate some of the barriers that have been identified to practicing EBM in a clinical setting (Ely 2005). The modest change that most residents reported to their current practice suggests the BEAR may be useful to assist clinical decision making. While the BEAR may be a useful first step for residents to help them gain familiarity with evidence-based resources, further research is required regarding the reliability of the answers found using the BEAR in addition to the sustainability of this intervention. Currently, we are assessing whether the BEAR has any impact on how recent graduates of the program answer clinical questions at the point of care.

Declaration of interest: All authors report no conflict of interest in the writing of this paper.

The authors would like to thank the College of Family Physicians of Canada Janus Grant program for partial funding of this project. Award Reference #G599001384.

The content of this manuscript has been presented as a poster at the Family Medicine Forum October 28, 2009, Calgary, Alberta, Canada.

Authors’ contributions

SA and VM were involved in data acquisition and interpretation. SR assisted with data analysis and interpretation in addition to manuscript revision. MA conceived of the study, and assisted in its design and co-ordination and final manuscript revision. TK assisted with the study design and co-ordination, data collection and drafted the initial manuscript. All authors read and approved the final manuscript.