Interns reflect: the effect of formative assessment with feedback during pre-internship

Abstract

Background

It is widely known that the opportunity for medical students to be observed and to receive feedback on their procedural skills performance is variable in the senior years. To address this problem, we provided our Pre-Intern (PrInt) students with “one-to-one” formative feedback on their ability to perform urethral catheterization (U/C) and hypothesized that their future practice of U/C as interns would benefit. This study sought to evaluate the performance and practice of interns in U/C 4–5 months after having received feedback on their performance of U/C as PrInt students.

Methods

Between 2013 and 2014, two cohorts of interns, (total n=66) who had received recent formative feedback on their U/C performance as PrInt students at Central Clinical School, were invited to complete an anonymous survey. The survey contained nine closed unvalidated questions and one open-ended question, designed to allow interns to report on their current practice of U/C.

Results

Forty-one out of 66 interns (62%) completed the survey. Thirty-five out of 41 respondents (85%) reported that the assessment with feedback during their PrInt term was beneficial to their practice. Thirty of 41 (73%) reported being confident to perform U/C independently. Eleven out of 41 respondents (27%) reported that they had received additional training at intern orientation. Nine of the 11 interns (82%) reported that they had a small, but a significant, increase in confidence to perform U/C when compared with the 30 of the 41 respondents (73%) who had not (p=0.03).

Conclusion

Our results substantiate our hypothesis that further education by assessment with feedback in U/C during PrInt was of benefit to interns’ performance. Additional educational reinforcement in U/C during intern orientation further improved intern confidence. Our results indicate that extra pre- and post-graduation procedural skills training, with feedback, should be universal.

Keywords:

• formative assessment with feedback
• performance and practice of new interns
• medical interns
• urinary catheterization
• pre-intern students

Introduction

The opportunity for senior medical students to perform core procedural skills, such as urethral catheterization (U/C), and to be observed¹ and receive feedback varies considerably.² Effective feedback is critical in procedural skills training; it allows the students to have insight into their clinical performance and it is the main component of formative assessment.³ While many authors have provided definitions for feedback, we chose Ende’s (1983) description of feedback as, “information describing students’ or house officers’ performance in a given activity that is intended to guide their future performance”.⁴ Receiving accurate feedback can assist in narrowing the gap between the students’ actual performance and the performance which is desired.⁵ In order for feedback to occur, the clinical task requires direct observation and assessment according to the specific performance criteria.^6,7

Senior medical students perform procedures in the hospital wards during clinical placements, and at the completion of their clinical attachment, they were generally given a “global assessment”.^8–10 However, safe performance is complex, requiring the application of many skills, including psychomotor skills, clinical judgment, communication skills, decision making, and patient-focused interactions.¹¹ Clearly, time constraints often prevent students from being directly observed⁷ or adequately supervised.⁶ Therefore, formal specific “one-to-one” feedback that enhances the learning process is often overlooked.¹²

We used a faculty-validated checklist for the formative assessment. This had previously been used by the students in this study for self-directed learning following urinary catheter training in their second year of medicine; it was also used as a reference point in their senior years. An emphasis was placed on Ericson’s model of “deliberate practice”.¹³ This involved the processing of information and the acquisition of skills with the type of feedback that reinforces positive aspects performance, which allows the learners’ insight into what they did well and what they could improve upon.

The socio-constructivist model of feedback is relative to our study as it contributes to the learning of the students by allowing conversation between the student and the assessor.¹⁴ To create good feedback, Vickery and Lake¹⁵ recommended that the learners need to be directly observed, the learners are provided with clear goals and outcomes, and that the assessors must be able to provide positive and negative feedbacks.¹⁵ Pendleton’s positive critique¹⁶ method of delivering feedback was used as a framework for the assessment component of this study to allow students to interact with the assessor and to construct meaning from their feedback information. This type of feedback allowed students to be immediately informed of their progress and allowed the students to reflect upon the aspects of their performance that needed to be improved.

It is generally assumed that interns are able to perform basic core skills, such as U/C under supervision when first entering the workplace.¹⁷ However, our previous study found that immediately prior to the commencement of internship, almost one-third (29%) of Pre-Intern (PrInt) students had never performed male U/C.¹⁸ Similarly, a recent UK study found that 20 out of 50 doctors (40%) had not performed male catheterization as medical students.¹⁹ Thomas et al reported that 6% (51/864) of urological consultations from July 2006 to June 2007 were related to U/C morbidities, and 74% (38/51) of these were intern related, with most (73%; 28/38) occurring during the first 6 months of internship.²⁰ A follow-up study by Sullivan et al that involved intern education at the start of the intern year found that male U/C-related morbidities that were attributed to new interns during their first 6 months fell by 27%, with 63% reporting confidence in performing male U/C compared with 35% prior to the program.²¹ Another study involving medical student education in U/C found that knowledge could be retained from skills laboratory training (with feedback) to real-life clinical situations for up to 6 weeks.²² While both studies reported the positive effect of education, neither had reported on the effects of assessment with feedback on early intern performance in U/C.

Our study sought to evaluate participants’ practice and performance of U/C as new interns following formative assessment in U/C performance, using Pendleton’s method of positive feedback¹⁶ immediately prior to graduation.

To the best of our knowledge, there are no recent data that report on the effects of “one-to-one” feedback in U/C in PrInt and its subsequent impact on the performance capabilities in early internship.

Context

The participants in this study were new interns who had completed a 4-year graduate entry medical degree at Sydney Medical School. All were students at Central Clinical School (CCS), which is attached to Royal Prince Alfred Hospital (RPA). While some remained as interns at their original student hospital, RPA, most were dispersed to regional, interstate, and international hospitals. Procedural skills training in U/C had been provided to the CCS medical students in years 2 and 3, and in PrInt, which is the final stage of medical student training immediately prior to internship. The duration of PrInt term was 8 weeks during 2012–2013.

Methods

PrInt assessment with feedback

Two cohorts of final-year PrInt students (n=66) at CCS were assessed and given individual feedback on their performance of male U/C as part of their preparation for internship between 2012 and 2013.

Study instruments

Our study instruments for providing assessment with feedback were a part-task training model to assess students’ psychomotor skills and knowledge in U/C and faculty-validated content referenced performance criteria checklist.

Part-task trainer

We used a part-task trainer as neither was it appropriate to use a real patient, nor was it financially viable to use an integrated procedural performance instrument (IPPI).¹¹ We felt that we could achieve a level of authenticity with the use of assessment protocols.²³ We used a faculty content-based validated checklist¹⁸ to facilitate direct observation. The checklist was developed by clinicians at Sydney Medical School. Its purpose is to act as an educational guideline for students’ self-directed learning. The performance criteria ranged from 1 to 26 (which included all criteria that are involved in catheterization, such as obtaining consent for the use of aseptic technique).¹⁸ The assessment criteria were a binary “yes” or “no” score.¹⁸ Assessors were senior clinicians who had been trained in the use of the checklist and the theory of Pendleton’s method¹⁶ of providing feedback (Table 1). The assessment took place over 15 minutes per student, followed by 5 minutes of one-to-one feedback; which was modelled on Pendelton’s method.¹⁶

Table 1 Feedback model

1. The learner states what was done well.

2. The observer(s) states what was done well.

3. The learner states what could be improved.

4. The observer(s) states how it could be improved.

Note: Use of the of feedback model was adapted from Pendleton’s positive critique method.¹⁶

Follow-up

The follow-up study took place between 2013 and 2014, when (n=66) interns who had received assessment with feedback as PrInt students at RPA were invited to complete an anonymous electronic survey questionnaire. This consisted of nine closed unvalidated questions and one open-ended question, which were compiled in consultation with senior clinicians. The questions included a number of factors that may have influenced interns’ current U/C practice including demographics, such as gender location, additional education, interns’ confidence with independent performance, frequency of performance, and the benefits associated with U/C feedback during PrInt.

Statistical analysis

All questionnaire results were recorded as categorical variables and presented as percentages. Chi-square tests were used to determine any differences between those who attended their parent hospital RPA, as well as U/C performance by gender. We considered p<0.05 to be statistically significant. Because of small numbers, and to perform chi-square testing, we combined responses namely: for “Intern success in U/C performance”, “Sometimes” and “Never” were combined. For “Intern confidence by hospital” and “Intern confidence to perform by additional training”, “Not very” and “Somewhat” were combined. Responses were missing for the following questions: one from question 1, three from question 2, one from question 4, and one from question 7. All statistical analyses were performed in Statistical Analysis System (SAS) Version 9.3.

Qualitative data analysis

The final question was open ended to elicit further comments from interns on their experience and performance with U/C. Respondents’ free text responses were collated and thematically analyzed by the first author.

Ethics approval

Ethics approval was obtained from The University of Sydney Human Research Ethics Committee (protocol number 2012/1764). A statement was included in the survey notifying the participants of the study intentions, and return of the survey was taken as consent to participate.

Results

Over a period of 2 years, a total of 41 out of 66 (62%) interns who had participated in assessment with feedback as PrInt students completed the survey 4–5 months into their internship.

Of the 41 respondents, 21 (51.2%) were males, 17 (41.4%) were females, and 3 (7.4%) did not report their gender. Fourteen (36%) respondents reported that they were allocated to their “parent hospital”, RPA, and 25 of 39 (64%) respondents were allocated to other hospitals (two frequencies were missing). Most (35/41, 86%) respondents reported that “assessment with feedback” in PrInt was beneficial to their U/C practice as interns. Virtually all (39/41, 95%) respondents reported that they were “somewhat or very confident” to perform U/C on a patient and 30 of 40 (75%) respondents stated that they had successfully performed U/C insertion.

Performance by hospital

Interns who were allocated to their original teaching hospital (RPA) reported a statistically higher frequency of independent performance of U/C, 13 of 14 (93%), when compared with 15 of 24 (63%) of their cohort peer interns who were students at RPA, but allocated as interns to other hospitals (p=0.04).

There was a non-significant trend for higher confidence in performing U/C among interns who were allocated to their original teaching hospital, RPA. Eight of 14 (57%) interns reported a higher level of confidence to perform U/C, when compared with 10 of 25 (40%) of interns who were attached to other hospitals (p=0.30). Interns at RPA also reported a higher, but not significant, success rate (12/14, 86%) when compared with their cohort peers who were allocated to other hospitals (17/24, 71%; p=0.30).

Performance by gender

Female interns reported a small, but not significantly higher, frequency of independent performance (13/16, 81%) when compared with male interns (15/21; 71%; p=0.49). However, 11 of 21 (52%) male interns reported to be more confident in performing U/C when compared with 7 of 17 (41%) female interns (p=0.79).

Male interns also reported a far higher frequency of performing U/C on male patients, with 7 of 21 (33%) having inserted more than 5 male U/C catheters, while 1 of 17 female (6%) interns performed U/C on a male patient.

Confidence by additional training in U/C

Eleven of 41 (27%) respondents reported that they had received additional training at intern orientation. Nine of these 11 interns (82%) reported that they were significantly more confident to perform U/C, when compared with the 30 of 41 (73%) interns who had minimal or no additional training (p=0.03).

Qualitative data

Analysis of open-ended responses resulted in three themes, which interns felt were helpful in their performance, namely, tips from experienced clinicians, use of checklists, and further education at intern orientation.

Tips from experienced clinicians

Empirical learning theory asserts that individual learning depends on building, drawing upon one’s existing knowledge.²⁴ Interns reported that “tips from experienced practicing clinicians” added knowledge, as they speak from experience and offer insight on how to deal with challenges that were ahead of them: “tips from experienced clinicians were of benefit, as they offered insight into challenges that we may soon encounter.” [Intern: 16]

The use of a checklist

Interns found the checklist to be insightful into the procedural protocol: “The Supervised (assessment) session with checklist was useful, as it gave me insight as to what I should have been doing”. [Intern: 32]

A checklist allows for an objective step-by-step assessment of microcompetencies⁹ to which a student must recall and implement. This is useful as it can be used in both teaching and assessment.²⁵

Further education at intern orientation

Interns found that having further education and training in U/C during intern orientation provided additional confidence and complemented the educational benefits of assessment with feedback during PrInt. They also found education as motivation to perform U/C on patients. “I had one additional session during my orientation week, which complements what I did last year” (i.e., it was on the pelvic models) “it inspired me… and gave me confidence to perform, I have since inserted two Indwelling catheters both in men”. [Intern:4]

Discussion

We sought to investigate the impact of assessment with feedback during PrInt on interns’ practice and performance of U/C in early internship. While many authors have explored the effect of assessment with feedback, we were also able to draw further perspectives from our results and were given some insight into U/C performance patterns of new interns. The key findings included demographic information, intern performance frequencies, and gender and performance of male U/C.

Intern performance patterns

Those who did their internship in their original student teaching hospital reported a higher frequency of independent performance of U/C, when compared with their peers from the same teaching hospital, but who were allocated to other hospitals as interns. Familiarity with their original teaching hospital procedures and staff may partially explain this. Alternatively, it may be that RPA students had more teaching and training on this procedure during orientation than other hospitals. Interns who were allocated to their original student hospital also reported significantly (p=0.03) higher confidence and success rate in U/C performance, when compared with interns who were allocated to other hospitals.

Gender specificity

Unsurprisingly, we found U/C performance patterns were “gender specific”,²⁶ with male interns reporting a far greater frequency of performing male catheterization, compared with females. These results validate a previous report of gender specificity, in that male interns receive more experience in male-specific skills.²⁶ This observation of gender specificity may also be explained by the policy of some hospitals that male U/C is done by interns, rather than nursing staff.

Education

Further training on U/C during internship orientation was inconsistent across different hospital sites. However, we found that additional training during orientation did improve interns’ confidence to perform U/C. Our qualitative findings indicated that interns who were provided with further education at orientation in the intern year reported that it provided both additional confidence and motivation to perform U/C. This supports a recent study by Sullivan et al, who reported that a new intern training program in U/C at the start of the intern year resulted in an increase in confidence in performing U/C, from 35% before training to 63% after the training program (p<0.05).²⁷ Although mandatory intern orientation and education are now part of an Australian national training program,²⁸ the content and consistency of U/C training in individual hospitals are unknown.

The use of checklists

Interns reported favorably on the use of a checklist, in that it gave insight into what they “should” have been doing. Checklists have been developed in some programs as an attempt for a standardized curriculum to teach and evaluate procedures.²⁹ When checklists are combined with prompt feedback, students are made aware of what they “know and don’t know”, which focuses on learning.³⁰ However, it must be noted that checklists also have drawbacks, in that the steps to perform a procedure are not differentiated between the important and the less important steps, and sometimes not all steps are essential for the successful completion of a procedure.³¹

Tips from experienced clinicians

Experienced clinicians who are experts in their field can provide valuable tips to students. The value of their expertise can be transmitted to the patient by the work of others in the form of training other clinicians.³² As stated by Irby, “experienced clinicians can also act as role models³³ for students by helping them to establish their identity”, as students learn by imitations and observation of the clinicians that they respect.³⁴

The significance of this study

The results from this study support the assessment with feedback in the theoretic framework of Pendleton’s method,¹⁶ by allowing PrInt students to reflect on their performance and allowing them to be aware of their performance strengths and limitations. If this was implemented in all clinical schools, the threat to the safety of the patients would be diminished.

Study limitations

The response rate to our survey was only 62%, despite one initial and two reminder e-mails; therefore, selection bias cannot be ruled out. We performed multiple subgroup analyses (gender, hospital, and additional training); despite the small sample size, this along with the absence of a control group must be taken into account when considering our conclusions.

Although all the initial participants attended the same clinical school as medical students (RPA), the main focus of this research relied on their experience as interns and the majority were placed at a variety of city, rural, and international hospitals. They reported an inconsistency of additional training during early internship, which led to our main outcome being confounded. Nevertheless, we consider this finding to be of significance as it indicated the need for assessment with feedback in pre-internship.

We utilized a part-task training model to assess students’ psychomotor skills and knowledge in U/C as it was not appropriate to use real patients, nor was it financially viable to employ actors to utilize an IPPI,¹¹ which would have been useful for nontechnical skills. We also acknowledge that assessment and feedback on a part-task trainer cannot replace the complexities of a real-life situation in a busy hospital ward. However, our aim was to seek standardization and technical proficiency in U/C, thus enabling direct observation, assessment, and feedback prior to our students entering internship.

Conclusion

Our results highlight the value of direct observation, assessment, and feedback on performance of U/C for PrInt students. This additional training ensures that new interns enter the workplace with an updated assessment of their ability to perform this common procedure. Despite the benefit of additional training during intern orientation, this was inconsistent with many interns receiving little or no further education at orientation.

Definitions

Intern training program

A period of 47 weeks of mandatory, supervised, work-based clinical training that includes medicine, surgery, and emergency medical care terms to meet regulatory requirements. The program also includes orientation, formal and informal education sessions, and assessment with feedback, and it may be provided by one or more intern training providers. Also called postgraduate year 1 (PGY1) (Australian Medical Council).²⁸

Global assessment

Assessment that requires the assessor to make an overall judgment on a procedure rather than a step by the assessment of microcompetencies.⁹

Acknowledgments

We would like to thank the interns who contributed to this study.

Disclosure

The authors report no conflicts of interest in this work.