Using patients’ charts to assess medical trainees in the workplace: A systematic review

Abstract

Objectives: The objective of this review is to summarize and critically appraise existing evidence on the use of chart stimulated recall (CSR) and case-based discussion (CBD) as an assessment tool for medical trainees.

Methods: Medline, Embase, CINAHL, PsycINFO, Educational Resources Information Centre (ERIC), Web of Science, and the Cochrane Central Register of Controlled Trials were searched for original articles on the use of CSR or CBD as an assessment method for trainees in all medical specialties.

Results: Four qualitative and three observational non-comparative studies were eligible for this review. The number of patient-chart encounters needed to achieve sufficient reliability varied across studies. None of the included studies evaluated the content validity of the tool. Both trainees and assessors expressed high level of satisfaction with the tool; however, inadequate training, different interpretation of the scoring scales and skills needed to give feedback were addressed as limitations for conducting the assessment.

Conclusion: There is still no compelling evidence for the use of patient’s chart to evaluate medical trainees in the workplace. A body of evidence that is valid, reliable, and documents the educational effect in support of the use of patients’ charts to assess medical trainees is needed.

Background

One of the challenges of teaching medical trainees is to choose an assessment method that is directed to enhance learning in addition to assessing clinical competence. Workplace-based assessments allow trainees to continually gather evidence of learning and formative feedback (Setna et al. 2010). Chart-stimulated recall (CSR), which was developed as an assessment tool for emergency medicine physicians (Munger et al. 1982) or its variant, case-based discussion (CBD), is a method proposed to assess curricula competencies.

Stimulated recall is a rigorous tool that allows the assessor to draw, identify and explore trainee’s thinking and clinical reasoning (Schipper & Ross 2010). During the encounter of CBD or CSR, assessors converse clinical cases with the trainees, using patient’s chart and notes to facilitate discussion, to assist in the verification and clarification of data and to aid in analyzing the clinical reasoning behind the diagnostic tests utilized and the management decisions taken. The systematic assessment of the encounter is coupled with an instant, structured, and contextual feedback. Charts are better selected by the assessor before the interview to identify key elements to develop case-specific interview questions that analyze the trainee’s clinical reasoning and judgment skills. A structured interview is used to elicit the trainee’s rationalization and determinants for data gathering, problem solving, and patient management. Work-based assessment using the patient’s chart attends to the base of Miller’s pyramid: “knows” and “knows how”, which conceptualizes the recall of knowledge and the application of the knowledge to problem solving and clinical decisions, respectively (Miller 1990; Norcini & Burch 2007). In the literature, CBD and CSR evidence are presented as separate terms, but they are in fact synonymous; as such, in this paper, both terms will be treated interchangeably. To our knowledge no systematic review combined both CSR and CBD to evaluate the evidence of using them as an assessment tool to enhance learning and assess clinical competencies of medical trainees.

Table

Practice points

Evidence supporting the use of CBD or CSR for the assessment of medical trainees is derived from subjective reports and non-comparative studies. Preliminary evidence exists for the reliability and validity of using patients’ chart to evaluate medical trainees; yet a comprehensive validity study based on a unified validity framework is needed. While both assessors and trainees acknowledge the assessment tool for its role in facilitating learning, identifying learning needs, and generating learning objectives, further research is needed to evaluate objectively the effect of the assessment tool on learning, behaviors, and patients’ outcomes. Lack of assessor training in the process is a major barrier to carrying out assessment of medical trainees using patients’ charts. Standardization of the scoring scale and the feedback framework and calibrating faculty members, as assessors are essential to the successful application of the assessment tool.

Objectives

1. To summarize and critically appraise existing evidence on the use of CSR or CBD as an assessment method for medical trainees.

2. To address the strengths and limitations of using CSR or CBD as an assessment tool in medical education.

Methods

Search strategy

The literature search was performed in April 2014. Published studies on the use of CSR and CBD, as assessment methods were identified using both manual and electronic search strategies. Only papers published in English were included. This search was applied to MEDLINE (1966 to April 2014), EMBASE (1980 to April 2014), CINAHL (1982 to April 2014), PsycINFO (1806 to April 2014), Educational Resources Information Centre (ERIC), Web of Science (1900 to April 2014), and the Cochrane Central Register of Controlled Trials (issue 4, 2014). Additional citations were sought by hand-searching the reference list of retrieved articles. A combination of text words and exploded medical subject headings were used to maximize the volume of literature retrieved. We used the Boolean search term “or” to explode and map the medical subject headings and the Boolean search term “and” to combine the searches identified. Outcome was not specified in our search to capture all potential studies evaluating the use of CBD and CSR to assess medical trainees.

Study selection criteria

Two reviewer authors (HA and RA) independently searched for and evaluated identified articles for eligibility for this systematic review. Criteria for the inclusion of studies were based on four components of the PICO format: (1) population: trainees in all medical specialties; (2) intervention: CBD or CSR as the only intervention used and not in combination with other assessment methods; (3) comparison: any other assessment tool or no comparison; and (4) outcome: we considered all outcomes for this review (Richardson et al. 1995). Further, all studies presenting original quantitative or qualitative assessment were eligible for inclusion. Studies were excluded if (1) participants were practicing physicians, nurses, or allied health trainees; (2) CSR or CBD was used in combination with other assessment tools, and (3) only the construction of the assessment tool was described and not its test administration and validity evidence.

Data abstraction and study review

Data abstraction was performed independently by two reviewers (HA and RA) using a data abstraction form that was developed and iteratively refined by two authors (HA and RA). Any discrepancies in extracted data were resolved by consensus.

Two reviewers (HA and RA) independently assessed the methodological quality of individual studies. Any disagreements were resolved by discussion, consensus, and consultation with a third reviewer (SS).

The methodological quality of the studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI) for education research, which includes 10 items that reflect six domains of study quality: study design, sampling, type of data, validity, data analysis, and outcomes (Reed et al. 2007). Messick’s unified validity framework was used to evaluate the validity of the assessment method (Messick 1989). The evaluation of outcomes was based on Kirkpatrick’s hierarchy of evaluating outcomes implementing an educational tool (1967).

Results

Study identification

Sixty-one articles were identified by our initial search strategy. Only seven articles (Bodgener & Tavabie 2011; Johnson et al. 2011; Mitchell et al. 2011; Jyothirmayi 2012; Brittlebank et al. 2013; Eardley et al. 2013; Mehta et al. 2013) were eligible for this review (Figure 1).

Figure 1. Summary of study selection process.

Study characteristics

Four of the included studies were qualitative studies that utilized questionnaires and semi-structured interviews to address the educational impact of using the patient’s chart for assessment of medical trainees in the workplace (Bodgener & Tavabie 2011; Johnson et al. 2011; Jyothirmayi 2012; Mehta et al. 2013). The remaining three articles were observational non-comparative studies (Mitchell et al. 2011; Brittlebank et al. 2013; Eardley et al. 2013). Articles included studied postgraduate trainees who were enrolled in oncology (Jyothirmayi 2012), family medicine (Mitchell et al. 2011), surgery (Eardley et al. 2013), pediatric (Mehta et al. 2013), psychiatry (Brittlebank et al. 2013), and more than one training programs (Bodgener & Tavabie 2011; Johnson et al. 2011).

Current evidence for the effectiveness of CSR or CBD

The evaluation of the domains of study design, sampling including response rate, and type of data are presented in Table 1.

Table 1. Characteristics and outcomes of included studies.

			Validity			Outcomes
Study (Specialty)	Methods	Participants (Response rate)	Internal structure	Content	Relationship to other variables	Reaction	Learning	Behavior	Patient Outcome	Challenges
Bodgener & Tavabie (2011) General practice setting	1-Questionnaire 2-Semi-structured interviews	Five trainees (100%) 30 trainers (78%) Five trainers five trainees	No No	No No	No No	Yes Yes	No No	No No	No No	• Case selection • Skills of giving feedback • Purpose of assessment: consultation or discussion
Brittlebank et al. (2013) Psychiatry	Prospective observational	220 trainees 574 CBDs	No	No	High correlations with peer assessment and presentational assessment (p < 0.001). Weaker correlations with DOP (p < 0.05). No correlation with PS.	Yes	No	No	No	No
Eardley et al. (2013) Surgery	Retrospective observational using trainee’s portfolio	17,904 assessments completed by 2736 trainees	No	No	There was no apparent relation between the stage of training and the summary rating given.	No	No	No	No	No
Johnson et al. (2011) All specialties	Questionnaire	42 trainees and 36 assessors	No	No	No	Yes	No	No	No	• Time constrains • Difficult grading • Arranging discussions
Jyothirmayi (2012) Oncology	CBD transcripts and semi structured interviews	Five consultants and five trainees	No	No	No	Yes	No	No	No	• Time constrains • Inadequate training in the process • Value of scoring scale
Mehta et al. (2013) Pediatric	1-Online questionnaire 2-Interview	26 trainees Nine trainees	No No	No No	No No	Yes Yes	No No	No No	No No	• Time • Choice of the case • Attitude and skill of the assessor • Suitable environment
Mitchell et al. (2011) Family medicine	Retrospective observational study using the trainee’s e-portfolio database	75,580 assessments completed by 1646 trainees (other assessment methods were included but evaluated separately)	No	No	Significant No association between average score and training difficulties of all types (OR = 0.39, 95% CI 0.21–0.72)	No	No	No	No	No

CBD, case-based discussion; CI, confidence interval; DOP, direct observation of procedural skills; OR, odds ratio; PS, patient satisfaction.

Evidence of validity

We evaluated each study using three sources of validity evidence including internal structure, content, and relationship to other variables from Messick’s unified validity framework. We considered consequential validity evidence together with outcomes in the ensuing section (Table 1). The number of encounters needed to achieve a sufficient level of reliability varied across studies. While Brittlebank et al. (2013) reported that four sessions were required to achieve a reliability of 0.8; 12 sessions were needed to attain a reliability of 0.7 in the study by Johnson et al. (2011). The relationship between the tool’s content and the construct measured (content validity) was not evaluated in any of the studies. Three of the included articles examined the relationship of the assessment tool to other variables (Mitchell et al. 2011; Brittlebank et al. 2013; Eardley et al. 2013). Mitchell et al. (2011) concluded a significant association between the average score and training difficulties of all types (OR = 3.9, 95% confidence interval [CI] = 0.21–0.72). Eardley et al. (2013) found no evident relation between the stage of training and the summary rating given. Further, Brittlebank et al. (2013) ascertained a high correlation between CBDs, peer assessment, and presentational assessments (p < 0.001). In the observational study by Brittlebank et al. (2013), a weaker correlation was found between CBDs and direct observed procedures (p < 0.05), while no correlation was established with the patient satisfaction assessment tool.

Consequential validity and outcomes

The educational impact of the assessment tool was mostly measured by surveying trainees and assessors through questionnaires and interviews that addressed their opinions on the tool and their experience with its implementation. Both assessors and trainees expressed high levels of satisfaction with the tool (Brittlebank et al. 2013; Mehta et al. 2013). Further, participants generally agreed that the tool provided evidence of learning (Johnson et al. 2011; Jyothirmayi 2012), identified learning needs (Bodgener & Tavabie 2011; Johnson et al. 2011) and generated learning objectives (Bodgener & Tavabie 2011; Mehta et al. 2013). Moreover, there was compelling evidence that the tool facilitated trainees to demonstrate clinical reasoning and knowledge around the case (Bodgener & Tavabie 2011; Johnson et al. 2011). Conversely, there was no consensus on the use of the tool to assess performance (Bodgener & Tavabie 2011; Mehta et al. 2013). An integral element of the process was to give feedback. Feedback was perceived to positively influence learning and decision-making skills (Bodgener & Tavabie 2011; Mehta et al. 2013). Time, choice of case, the skills of the assessor in giving feedback, and the environment in which the encounter takes place were factors identified to influence the educational value of giving feedback. Challenging cases, assessors trained in the process with positive attitude, ability to give constructive feedback, and dedicated place for the encounter stimulated more valuable and useful feedback (Bodgener & Tavabie 2011; Mehta et al. 2013).

Limitations in the use of the assessment methods

Time was reported as a limitation of the assessment tool by both assessors and trainees (Johnson et al. 2011; Jyothirmayi 2012; Mehta et al. 2013). Brittlebank et al. (2013) reported that it took 25 ± 16 min to complete the session, while two-thirds of the sessions took less than 20 min in the study by Johnson et al. (2011). In addition, inadequate training in the process and the skills of the assessor to give feedback were major barriers in conducting the assessment of medical trainees using a patient’s chart (Bodgener & Tavabie 2011; Johnson et al. 2011; Jyothirmayi 2012; Mehta et al. 2013). Furthermore, assessors questioned the value of the scoring system used to evaluate the session mainly as a result of different interpretations of the scoring scales that would precipitate difficult grading (Johnson et al. 2011; Jyothirmayi 2012). Case selection was another limitation acknowledged by both trainees and assessors, who chose the case on one end and the type of case on the other end. Trainees either chose challenging and complex cases to enhance the educational value of the session or cases where they know they performed well to score higher (Bodgener & Tavabie 2011; Mehta et al. 2013).

Discussion

Assessment of medical trainees in the workplace is fundamental to their development and ongoing learning (Singh & Modi 2013). Although there is a good evidence that work-based assessment tools are reliable and valid (Setna et al. 2010), there is still no compelling evidence from this systematic review for the use of patient’s chart to evaluate medical trainees in the workplace to make summative and high-stakes decisions. Further studies are needed to refine our understanding of how patient chart reviews contribute to the overall assessment system in the workplace.

Patients’ chart as an assessment tool is used to assess the clinical competence of practicing health physicians and other allied health professionals. Jennett et al. (1995) used both chart audit and CSR to assess the clinical competence of 20 practicing family physicians. Chart stimulated recall was found to expound on a broader range of information that could be used to assess physician’s diagnostic, investigative, and treatment skills and the clinical reasoning behind those skills. Mynors-Wallis et al. (2011) conducted a pilot study using CBD as an assessment method for revalidation of 86 consultant psychiatrists. Eighty-five physicians described the process as useful in improving clinical practice. Further, all of the assessors and 87% of the assessed physicians agreed that CBD was a useful quality improvement tool for revalidation. Salvatori et al. (2000) reported a high correlation between supervisors’ ratings and the CSR’s interviewers’ global ratings of overall competence of practicing occupational therapists denoting concurrent validity.

The evidence supporting the use of CBD or CSR for the assessment of medical trainees emanated from subjective reports on their educational impact. Using patient’s chart to assess trainees is an opportunity to assess several skills, including clinical reasoning, decision making, medical knowledge, and patient care (Emsden & Thomson 2010). Besides aiding the assessment of clinical skills, the tool can also be used to assess soft skills as communication skills, professionalism, and ethics that can mold the trainee’s competency (Singh & Modi 2013). Moreover, using patients’ chart for assessment of medical trainees is useful for all levels of training, challenging the advanced learners, and identifying those experiencing difficulties (Schipper & Ross 2010). Nevertheless, this contention is not supported by objective evidence for its use to assess postgraduate medical trainees. Norcini et al. (2011) outlined the criteria for good assessment which include validity, reproducibility, equivalence (different versions yield interchangeable results), feasibility, educational effect, catalytic effect (improve learning through results and feedback), and acceptability. Only subjective measures of the educational effect, feasibility, and acceptability could be ascertained from this review. Given the paucity of evidence available, the use of CBD or CSR needs further studies to affirm its role within assessment system in the workplace; moreover, its use to make summative and high-stakes decision requires additional validity evidence to be gathered.

Driessen and Scheele (2013) stated that “Only when we shift the emphasis in workplace-based assessment from assessment of trainee performance to the learning of trainees, will workplace-based assessment gain credibility in the eyes of trainees and supervisors”. Moreover, before challenging its validity, stimulated recall needs to adhere to strict methodological practices and standardization to translate the process into a reliable, valid and unbiased assessment of medical trainees. Further, assessors need to be trained on what to assess and the framework for giving feedback, as it is central to the educational impact of the structured formative assessment tool (Ramani & Krackov 2012). In addition, standardizing CBD or CSR should attend to the different interpretations of the assigned scoring scales, which tend to inflate the scores given in a summative rather than formative way (Johnson et al. 2011).

Future research is needed to further document validity evidence associated with CSR or CBD to assess the clinical competencies of medical trainees. These validity studies should be conducted within a comprehensive validity framework, such as Messick’s unified validity. In addition, it would be of immense value to investigate the impact of using CSR or CBD on changing practice behaviors and improving patient outcomes.

Notes on contributors

HEIDI AL-WASSIA, MD, is Assistant Professor, Department of Pediatrics at King Abdulaziz University, College of Medicine, Jeddah, Kingdom of Saudi Arabia.

ROLINA AL-WASSIA, MD, is Assistant Professor, Department of Radiation Oncology, College of Medicine, Jeddah, Kingdom of Saudi Arabia.

SHADI SHIHATA, MD, is Consultant Orthopedic Surgery, Department of Community Health Sciences, University of Calgary, AB, Canada.

YOON SOO PARK, PhD, is Assistant Professor, Department of Medical Education at the University of Illinois at Chicago College of Medicine, Chicago, IL, USA.

ARA TEKIAN, PhD, MHPE, is Associate Professor, Department of Medical Education, and Associate Dean for International Education at the University of Illinois at Chicago College of Medicine, Chicago, IL, USA.

Ethical approval

This study did not require ethical review from an institutional review board, as the study is a systematic review of the literature.

The publication of this supplement has been made possible with the generous financial support of the Dr Hamza Alkholi Chair for Developing Medical Education in KSA.

Declaration of interest: The authors report no declarations of interest.