Reaching the limits of mandated self-reporting: Clinical logbooks do not predict clerkship performance

Abstract

Background: Logbooks are used by US medical schools to evaluate curricular objectives and meet accreditation requirements, but research supporting their utility is conflicting.

Aims: The goal of our study was to examine the relationship between volume of clinical rotation experiences as reported in a logbook and clerkship grades within a longitudinal integrated clerkship.

Methods: We conducted a retrospective cohort study of third-year (clinical) medical students during academic year 2008–2009. We tracked student entries in a pocket-sized logbook (number of clinical encounters per clerkship, total number of exams, and procedures over the academic year). We performed correlation analyses between logbook entries and clerkship grades.

Results: We enrolled 36 students, who reported a total of 2992 encounters, 2262 exams, and 2342 procedures. Correlation coefficients between volume of clinical experience and clerkship grades were less than 0.4, indicating low correlation. We found borderline statistical significance for the Neurology, OB/Gyn, and Surgery clerkships. Sensitivity analyses showed little correlation between low-reporting activity and clerkship grades.

Conclusions: Even within an integrated longitudinal clerkship framework, our findings are consistent with previous studies showing a negligible relationship between logbooks as an educational process measure and how they relate to educational outcomes.

Introduction

Logbooks have been used in varied settings and formats to address curricular needs, to compare experiences across different sites (Connolly et al. 2010), and more recently to fulfill accreditation requirements (Connolly 2006). Defined as any method by which students document clinical experiences during their clerkships, logbooks may be helpful as guideposts for what students should be seeing, for topics not yet covered, and for engaging students in their learning agendas as adults. Logbooks can also provide quantifiable data with which faculty can measure alignment with curricular objectives (Rattner et al. 2001; Denton & Durning 2009), as stipulated by the Liaison Committee on Medical Education's (2011) ED-2 standard, applicable only to accredited US and Canadian medical schools. Both pedagogic principles and regulatory forces support the tremendous efforts institutions have undertaken to monitor student experiences.

Whether logbooks have achieved what they were intended to do, however, remains unproven. Their utility has been covered in a narrative review (Denton et al. 2006), which discussed the feasibility of logbooks, accuracy, utility to students and faculty, role in program evaluation, and relationship with performance outcomes. Denton suggested that the most pressing research agenda was to scrutinize the link between educational process measures, such as logbooks and educational outcomes. Only a few studies since then have examined the relationship between clinical experience and clerkship performance measures as proxies for clinical competence. One study showed no correlation between logged patient experiences during a surgical rotation and end-of-rotation exam or United States Medical Licensing Examination Step 1 scores (Neumayer et al. 1998). Another study showed that students directed toward 10 required cases performed better on an emergency medicine cognitive exam than those who took the standard approach to selecting patients (Lampe et al. 2008). More recently, a study of students on a neurology clerkship showed no correlation between the number of encounters recorded and clinical performance as measured by a global rating scale (Poisson et al. 2009).

We are unaware of any recent studies that examine the relationship between logbooks and clerkship grades across the third year (first clinical year) experience. Furthermore, we know of no literature that examines logbooks within the setting of longitudinal integrated clerkships (LICs). LICs represent a curricular framework developed in response to fragmentation and lack of assessment continuity due to multiple sites for clerkships. LICs are increasing in frequency in North America (Norris et al. 2009) and offer an opportunity to integrate clinical experiences and identify where content might be covered over the span of the year.

We have previously described our LIC, called the Principal Clinical Experience (PCE), whereby students undergo all their core clerkships (e.g., medicine, surgery) at a single site for the duration of the third year, instead of rotating among many different teaching hospitals for each block. This arrangement permits closer monitoring of student experiences and performance (Bell et al. 2008). At Beth Israel Deaconess Medical Center (BIDMC), clerkship directors and PCE mentors monitor students’ use of logbooks and encourage attention to content, exams, or procedures not yet experienced. This study sought to examine the relationship between the number of clinical encounters, as recorded in patient logs, and clerkship grades, both within rotations and across a longitudinal curriculum. We hypothesized that we would find a correlation between some, but not all, of the clerkships and reported experiences in those clerkships.

Table

Practice points

Logbooks have been widely used to track clinical encounters and ensure completion of curricular objectives. LICs enable medical students to spend their entire clinical year at a single site and are designed to mitigate fragmentation of education and assessment that may occur with block rotations at multiple sites. Within an LIC, volume of clinical experience had minimal correlation with clerkship grades. Educators should be cautious about ascribing value to logbooks as correlates or predictors of educational outcomes.

Methods

Study setting

At Harvard Medical School (HMS), medical students are assigned to one of four teaching hospitals for their PCE. For the PCE, students attend and take turns leading case conferences, meet regularly with mentors, follow ambulatory patients longitudinally, and may participate in a host of electives (e.g., simulations, writing workshops, computer-based modules) (Bell et al. 2008).

Data collection

In academic year 2008–2009, a PCE requirement was completion of a pocket-sized handwritten patient log. The patient log had three sections: (1) patient encounters (e.g., 72-year-old male with dyspnea and congestive heart failure), (2) examinations (e.g., fundoscopy, breast exam), and (3) procedures (e.g., peripheral IV, suturing). For each entry, we asked students to document the date of the event (encounter, procedure, or exam) and a description, if relevant. We collected the logs at the end of the academic year. A research assistant counted the patient encounters for each rotation, the procedures for the year, and the number of examinations for the year.

HMS clerkship grades are determined by a combination of faculty and resident evaluations of student performance in their clinical duties and standardized end-of-rotation knowledge tests. Clerkships may also use other assessments in the determination of the grade, such as oral exams or online modules. Possible grades are unsatisfactory (U), satisfactory (S), honors (H), and high honors (HH). Within the Medicine clerkship only, students can receive honors plus (H+), which is an off-the-transcript designation of performance intermediate between an H and HH. We collected clerkship grades from the clerkship directors.

Analyses

We generally treated individual clerkship grades as ordinal variables. We assigned a numeric value to each grade (U = 1, S = 2, H = 3, H+ = 3.5 for Medicine grades, HH = 4) and calculated an unweighted mean representing overall third-year performance. We used Spearman correlations to determine the relationship between the number of encounters in each clerkship and the respective clerkship grade. We used Pearson product-moment correlations to analyze the relationship between the overall third-year performance and (1) total number of encounters, (2) total number of exams, and (3) total number of procedures.

We performed two sensitivity analyses. We examined the relationship between an S or U grade in a given clerkship and the encounter number in that same clerkship. We also used simple regression analysis to examine the relationship between low reporting (reporting a number of encounters that was in the lowest quartile within a given clerkship), and the grade for that clerkship.

We performed all analyses using Stata/IC 8 (College Station, TX). The project was approved by the institutional review board of Harvard Medical School, and we received waiver of consent for subjects.

Results

In academic year 2008–2009, 39 students were enrolled in the BIDMC PCE. We excluded from the study three students who did not complete the academic year. Students reported an aggregate of 2992 encounters, 2262 exams, and 2342 procedures.

The Medicine clerkship, followed by Surgery, had the highest average number of recorded encounters. All but one clerkship included at least one student who reported zero encounters; however, all students reported more than 20 total encounters, exams, and/or procedures. All correlation coefficients between each clerkship and the corresponding number of encounters were less than 0.4, indicating low correlation (Table 1). Three clerkships (Neurology, OB/Gyn, and Surgery) had p-values that showed borderline statistical significance.

Table 1  Numbers of clinical events and correlation with grades, by rotation and overall

	Mean	SD	Range	Mean grade	Correlation coefficient with grades^a	95% Confidence intervals^b	p-value
Encounters by rotation
Medicine	20.4	10.9	0–71	3.5	0.28	−0.056 to 0.555	0.10
Neurology	5.7	4.7	0–26	3.1	0.34	−0.001 to 0.606	0.051
OB/Gyn	11.1	10.3	0–57	3.6	0.33	−0.005 to 0.604	0.054
Pediatrics	11.4	6.5	0–32	3.4	0.17	−0.170 to 0.470	0.33
Psychiatry	9.0	7.6	0–44	3.6	−0.03	−0.356 to 0.301	0.86
Radiology	5.9	5.0	0–24	3.2	−0.03	−0.354 to 0.303	0.87
Surgery	13.7	20.0	4–114	3.4	0.33	−0.003 to 0.591	0.052
Overall events^c
Encounters	83.1	57.3	22–322	3.4	0.17	−0.169 to 0.471	0.32
Exams	64.6	20.7	21–106		−0.01	−0.339 to 0.328	0.97
Procedures	66.9	23.9	21–133		−0.01	−0.339 to 0.327	0.97

Notes: ^aSpearman correlation for individual clerkship grades and Pearson correlation for overall third-year grade.

^bConfidence intervals based on Fisher's transformation.

^cOne student reported exams and procedures in a non-standardized format and was therefore not included in the calculation of mean exams and procedures.

In post hoc calculations, the number of S or U grades was too low (range 1–6 per clerkship) to conduct meaningful analyses. We also found no statistically significant relationship between students reporting an encounter number in the lowest quartile and their grade in that particular clerkship, although comparisons in the neurology clerkship approached statistical significance (p = 0.053).

Discussion

Within a longitudinal third-year curricular framework, we found a marginal association between reported clinical experience and clerkship grades in Neurology, OB/Gyn, and Surgery. No relationship was detected for all clerkships viewed as a whole. We also found a negligible relationship between low reporting behavior (either due to lack of diligence or limited clinical exposure) and clerkship grades.

We had intuited that a greater number of clinical exposures ought to have a more substantial and generalized impact on performance; practice is, after all, a cornerstone of clinical competence (Neufeld & Norman 1985). Experience should lead to greater facility in the diagnostic process, which is an important factor in determining clerkship performance (Wimmers et al. 2008). Furthermore, more patients may result in greater exposure to physician supervisors and therefore more opportunity for feedback to hone the skills of doctoring, as is supported by principles of deliberate practice (Ericsson 2004). This latter point may explain our findings of borderline statistical significance within the OB/Gyn and Surgery clerkships, where students receive close and prolonged supervision in the operating room setting. Nevertheless, our results in Neurology are not strong enough to dispute the study showing no relationship between logged clinical experience and average global ratings of clerkship performance in neurology (Poisson et al. 2009). Those authors’ findings persisted when comparing experience in specific conditions to performance on disease-specific questions on a standardized test, further indicating a discrepancy in how experience translates into performance.

Nevertheless, we recognize that clerkship grades are influenced by a multitude of factors and may not be the most sensitive metrics for the impact of clinical experience. As shown in an elegant analysis, Wimmers et al. (2006) demonstrated that while the number of patient encounters had a statistically significant (albeit modest) correlation with clinical evaluations, quality of supervision had a more direct effect on end-of-year exam scores.

The limitations of this study include reporting bias. Multiple studies show only moderate concordance between logbook encounters and actual encounters (Ferenchick et al. 2009), with underreporting being most common (Patricoski et al. 1998; Dolmans et al. 1999; Lee et al. 2002). Our numbers of encounters are comparable to previous studies in that we saw a broad range of use. It is also possible that our study was constrained by the use of paper-based logbooks rather than electronic systems; one prior study showed a higher number of encounters recorded using handheld computers (Marshall & Sumner 2000), although another showed low satisfaction with electronic logbooks (Penciner et al. 2007). We did not examine time of year as a covariate (e.g., students logging fewer encounters due to fatigue or logging more encounters due to experience with the system) but suspect that type of rotation contributes much greater variance than would change in student behavior over time. Additionally, our clerkship grades display positive skew and are also influenced by factors that have a less direct relationship with experience, such as professionalism, communication skills, skill in write-ups, and shelf exam performance. Finally, our sample size was small, our study may be underpowered, and it represented a single academic year at a single institution.

Conclusions

Our results add to the evidence showing that reported volume of clinical experience plays a minimal role in summative assessments of clinical performance. Based on our findings, medical educators should remain suspect about the value of logbooks in predicting clerkship performance not only on an individual clerkship, but also in an integrated longitudinal clerkship. Additionally, solely meeting expectations of volumes of types of patients and documenting these encounters may not be sufficient to address clerkship objectives. Further research on logbook utilization or newer logging technologies is warranted to identify how best to incorporate use of logbooks in evaluating student performance. Larger scale and longer term studies that include other variables known to affect clerkship grades may also be required to determine the relative contribution of experience to summative evaluations. These future studies may also help clarify whether student performance in clerkships with close supervision benefit from a higher volume of patients.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.