Pre-training evaluation and feedback improve medical students’ skills in basic life support

Abstract

Background: Evaluation and feedback are two factors that could influence simulation-based medical education and the time when they were delivered contributes their different effects.

Aim: To investigate the impact of pre-training evaluation and feedback on medical students’ performance in basic life support (BLS).

Methods: Forty 3rd-year undergraduate medical students were randomly divided into two groups, C group (the control) and pre-training evaluation and feedback group (E&F group), each of 20. After BLS theoretical lecture, the C group received 45 min BLS training and the E&F group was individually evaluated (video-taped) in a mock cardiac arrest (pre-training evaluation). Fifteen minutes of group feedback related with the students’ BLS performance in pre-training evaluation was given in the E&F group, followed by a 30-min BLS training. After BLS training, both groups were evaluated with one-rescuer BLS skills in a 3-min mock cardiac arrest scenario (post-training evaluation). The score from the post-training evaluation was converted to a percentage and was compared between the two groups.

Results: The score from the post-training evaluation was higher in the E&F group (82.9 ± 3.2% vs. 63.9 ± 13.4% in C group).

Conclusions: In undergraduate medical students without previous BLS training, pre-training evaluation and feedback improve their performance in followed BLS training.

Introduction

Basic life support (BLS) is a very important clinical skill for medical students. Due to limited cases and randomness of cardiac arrest and ethical consideration, it is not practical for each medical student to practice BLS skills on victims of cardiac arrest. Simulation-based BLS training provides a solution for effective training in these rare events.

Evaluation and feedback are two of factors that could influence quality of simulation-based medical education (Issenberg et al. 2005). The time when evaluation and feedback were given might contribute to their different effects on learning. Usually, feedbacks were delivered during (concurrent feedback) BLS training and trainees were then evaluated after the BLS training was finished (post-training evaluation). It is demonstrated that concurrent feedback can promote clinical skill acquisition (Ende 1983; Schmidt et al. 1999). However, concurrent feedback during BLS training might impose a high-extraneous cognitive load on trainees and contribute to trainees’ development of dependence, which might impair feedback quality (Walsh et al. 2009). Evaluation is not only a tool to assess technical skills competency, the results in the evaluation also could be used as a source from which feedbacks derived. However, since the training was already ended, few of results in post-training evaluation could be delivered as feedbacks to current trainees and help them to improve their acquired skills unless further training was instructed. We suppose evaluation and feedback that were conducted before BLS training (pre-training evaluation and feedback) might have beneficial effect on medical student's BLS skill acquisition. This study is to investigate impact of pre-training evaluation and feedback on medical student's performance in BLS.

Table

Practice points

Pre-training feedback could highlight the key points of BLS for trainees and guide both instructors and trainees’ focus and attention in the coming training. Pre-training evaluation is an effective way not only to identify trainee's basic skill level, but also to reveal trainee's mistakes, which could be delivered as a pre-training feedback from that both instructors and trainees could learn. Group feedback originated from pre-training evaluation minimizes unnecessary time spent on correcting individual's similar mistakes and less interruption might occur during BLS training. Group feedback also provides trainees an opportunity to learn from each other and even competition might arise among students, which might stimulate students’ enthusiasm to learn and improve the quality and efficacy of training. More time might be spent in improving quality and efficacy of evaluation and feedback, rather than in training. The best integration of evaluation and feedback in BLS training should be investigated in the future.

Materials and methods

This prospective, randomized, single blind, controlled cohort study was approved by the Institutional Review Board at the West China Hospital of Sichuan University. A written informed consent was obtained from all participants. A summary of the study profile was shown in Figure 1.

Figure 1. A summary of study profile. C group, the control group; E&F group, pre-training evaluation and feedback group. BLS, basic life support.

Forty 3rd-year undergraduate medical students without previous BLS training experience were randomly (sealed envelope method) divided into two groups, C group (the control) and pre-training evaluation and feedback group (E&F group), each of 20.

Following a 45-min BLS theoretical lecture, the C group received 45 min BLS training and the E&F group was evaluated seriatim (video-taped) in a mock cardiac arrest prior to their BLS training (pre-training evaluation). In the C group, individualized concurrent feedbacks (both verbal and demonstration) were given during BLS training. In the E&F group, following 15 min of group feedback (both verbal and videos recorded in the pre-training evaluation) delivered by BLS instructors, a 30-min BLS training was then instructed. In the E&F group, the two instructors only organized the BLS training following the guidance of an American Heart Association (AHA) certified BLS CD-ROM for Students (2006) and did not give any individualized concurrent feedback during training. After BLS training, both groups were assessed with a 12-item multiple-choice written examination (Appendix 1) and evaluated one-by-one with one-rescuer BLS skills in a 3-min mock cardiac arrest scenario (video-taped) by one blinded reviewer (post-training evaluation). The mock cardiac arrest scenario was that the student was asked to do one-rescuer BLS for a sudden fell patient (Resusci® Anne Basic, Laerdal Medical Corporation, Stavanger, Norway) in a ward.

The BLS theoretical lecture included introduction of mechanism and skills demonstration of BLS via a PPT file and videos that followed the guideline of AHA (2005) (ECC Committee, Subcommittees and Task Forces of the American Heart Association 2005). BLS training was instructed following AHA certified BLS CD-ROM for Students (2006) and 10 manikins (Resusci® Anne Basic and SkillGuide™, Laerdal Medical Corporation, Stavanger, Norway) were used for BLS training in both groups. One Resusci® Anne Basic and SkillGuide™ was used in BLS evaluation in both groups. Two video cameras (Sony, Japan) were used to record details of trainee's BLS performance during skill evaluation. Four teachers, including one lecturer, two BLS training instructors, and one blinded reviewer, were AHA certified and played the same role in both groups.

After all videos (both in pre-training and post-training evaluations) were collected in random order, the score of BLS skills was evaluated by the blinded reviewer using a 23-item checklist. The checklist was designed and in co-operation with the two BLS training instructors to match the curriculum of the course and expected skill level. Each item was scored on a scale of 1 and a maximum is of 23. Poor performance or failing to perform the item was scored as 0. The checklist for BLS skill evaluation is shown in Appendix 2. Accurate (yes or no) rate of five discrete BLS skills, as effective mask ventilation, correct chest compression site, acceptable chest compression rate (90–110 compressions per minute), correct depth of chest compression (1.5–2 inches) and limited cardiopulmonary resuscitation (CPR) free intervals (<15 s) were evaluated, respectively, as well. The blinded reviewer was trained how to evaluate BLS skills via recorded video (not used for BLS training) according to checklist mentioned-above before this study.

Primary outcome measure and statistical analysis

The primary outcome was the improvement of score of BLS skills during post-training evaluation. A sample size of 30 was determined to identify a 15% improvement of overall band score of BLS skills during post-training evaluation with respect to a type I error of 0.05 and power of 0.9. The scores from the multiple-choice written examination and BLS skills evaluation were converted to a percentage and were compared between the two groups using Student's t-test and reported as a mean ± SD. Chi-squared test was used to analyze difference of accurate rate of five discrete BLS skills during post-training evaluation between the two groups. A p < 0.05 was considered to be statistically significant.

Results

No student has previous personal or professional experiences in BLS in all 40 enrolled medical students and all of them completed investigation protocol. The demographic data were shown in Table 1. Score of BLS skills in pre-training evaluation in E&F group is 40.2 ± 12.6% (95%CI, 22.2–60.0%). There was no significant difference of scores of the multiple-choice examination between the two groups (79.8 ± 9.2% in C group vs. 83.8 ± 6.3% in E&F group, p = 0.12). Higher score was observed in E&F group (82.9 ± 3.2%, 95% CI (80.0–88.9%)) in post-training evaluation, compared with C group (63.9 ± 13.4, 95% CI (41.1–88.9%)), p < 0.01. Except correct compression site, higher accurate rate of other four discrete BLS skills were observed in E&F group, p < 0.05 (Table 2).

Table 1.  Demographic data.

	C group	E&F group
Total number of students	20	20
Sex (male/female)	7/13	6/14
Age (years)	21.4 ± 0.7	21.6 ± 0.8

Note: Data are number or mean ± SD.

Table 2.  Scores from pre-training evaluation in E&F group, and the multiple-choice examination and post-training evaluation in two groups.

		C group		E&F group
		Mean ± SD (%)	95% CI	Mean ± SD (%)	95% CI	p
	Score of BLS skills in pre-training evaluation	–	–	40.2 ± 12.6	22.2–60%	–
	Multiple-choice examination	79.8 ± 9.2	66.7–93.0%	83.8 ± 6.3	75.0–91.7%	0.12
	Score of BLS skills in post-training evaluation	63.9 ± 13.4	41.1–88.9%	82.9 ± 3.2*	80–88.9%	<0.01
Five discrete BLS skills	Effective mask ventilation	70.0	–	95.0*	–	0.037
	Correct chest compression site	95.0	–	100	–	0.31
	Acceptable chest compression rate	55.0	–	85.0*	–	0.04
	Correct depth of chest compression	60.0	–	90.0*	–	0.03
	Limited CPR-free intervals	40.0	–	75.0*	–	0.03

Notes: C group, the control group; E&F group, pre-training evaluation and feedback group; BLS, basic life support; SD, standard deviation; 95% CI, 95% confidence intervals.

*p < 0.05 compared with C group.

–, there is no content.

Discussion

Feedback and evaluation are two of important characteristics in simulation-based medical education and they have variable influence on learning through their format, content, and programming. In this study, although individualized concurrent feedbacks were delivered to trainees during their training in the C group, better outcome was observed in the E&F group, in which group feedback was given prior to their BLS training. We presumed that multiple factors might contribute to this result.

The time of feedback and evaluation conducted was thought to be a contributor of the different outcomes in two groups. The time of feedback delivered has been shown to have different effects on motor learning (Xeroulis et al. 2007). Concurrent feedback is the feedback that people receive during their skills performance. It is demonstrated that concurrent feedback can promote clinical skill acquisition (Ende 1983; Schmidt et al. 1999). However, according to cognitive load theory, humans have a limited attentional capacity such that they can only focus on a limited amount of information simultaneously (Sweller et al. 1998; Schmidt et al. 1999; Xeroulis et al. 2007). Concurrent feedback may impose a high-extraneous cognitive load to both trainee and the instructor. For the trainee, the increased amount of cognitive processing required performing a task while receiving, interpreting, and responding to concurrent feedback may interrupt their attention on learning the moves required to solve the problem at hand (Sweller et al. 1998). Giving concurrent feedback may also place excessive cognitive demands on instructors since they are required to observe the trainee's performance while simultaneously providing feedback (Sweller et al. 1998). This extra work load for instructors may potentially disrupt their concentration and impair feedback quality. Terminal feedbacks are feedbacks given after skills performance. In this study, although feedback was given prior to BLS training (pre-training feedback) in the E&F group, it is per se a kind of terminal feedback, as it was given after pre-training evaluation was completed (Ende 1983; Schmidt et al. 1999). Terminal feedback is considered to be superior in promoting learning during simulation-based education (Xeroulis et al. 2007). Benefit of pre-training feedback is that errors during pre-training evaluation can be allowed to progress, so both instructors and trainees learn from those mistakes. Pre-training feedback might highlight the key points of BLS for trainees and guide both instructors and trainees’ focus and attention in the coming training. Consequently, quality and efficacy of training is improved. Thus, use of pre-training feedback as a learning tool for BLS training is more significant in the E&F group, compared with concurrent feedback instructed in the C group.

In addition, better results in the E&F group might be partially attributed to pre-training evaluation. It is a general assumption that “assessment drives learning” through so-called testing effect, a robust and independent phenomenon that was demonstrated to apply to a variety of test formats and levels of knowledge (Brazeau et al. 2002; McLachlan 2006; Roediger & Karpicke 2006; Schoonheim-Klein et al. 2006; Walsh et al. 2009; van der Vleuten 1996). Evaluation is not only a tool to assess technical skills competency, but also to provide an opportunity (feedbacks) for instructors to know either flaws of their own instructions or students’ shortcoming in training. And these feedbacks might help instructors to focus more on helping trainees in difficult points and improve the quality of their instructions in the future. But the traditional programming BLS course (evaluation is usually arranged after training is finished) limits the beneficial effects of feedbacks from post-training evaluation on the current trainees. Few of those feedbacks derived from post-training evaluation could be delivered to current trainees since the training was already ended. Few of trainees’ acquired skills could be improved without repetitive practice unless further training was instructed. Compared with traditional post-training evaluation in the C group, pre-training evaluation was conducted before BLS training in the E&F group. Pre-training evaluation is an effective way not only to identify trainee's basic skill level, but also to reveal trainee's mistakes, which is quite useful to be delivered as a pre-training feedback (terminal feedback per se) to learning from both for instructors and trainees. Compared with limited effects of feedbacks derived from post-training evaluation on trainees’ acquired skills in the C group without further practice, feedback originated from pre-training evaluation could help both the instructors and trainees to concentrate on those key points of BLS in the following training in the E&F group. As a result, quality of BLS training was improved and better performance observed in the E&F group.

Furthermore, objects to whom feedbacks were delivered could be another contributor for different results in the two groups. The object of individualized concurrent feedback in the C group was each individual trainee in the C group. Individualized concurrent feedback in the C group during BLS training might offer each trainee individualized instructive information related to his/her performance and is beneficial for improving their performance individually. However, unnecessary time might be spent in giving similar feedback to different individuals as different learners might make the same mistake. More interruption might occur during BLS training. Due to time limitation of the BLS training course, the quantity, quality, and efficacy of individualized feedback could be impaired consequently. In the E&F group, the object of feedbacks was the whole group. Group feedback originated from pre-training evaluation in the E&F group highlighted key points of BLS for trainees and minimizes unnecessary time spent on correcting individual's similar mistakes during BLS training. Therefore, less interruption might occur during BLS training in the E&F group and both instructors and trainees could focus more on training and practice. Moreover, group feedback also provides trainees in the E&F group an opportunity to learn from each other and even competition might arise among students, which might stimulate students’ enthusiasm to learn and improve the quality and efficacy of training. These might explain that less training time was spent in the E&F group, but better outcome was observed, compared with the C group. This result suggests that appropriately more time might be spent in improving quality and efficacy of evaluation and feedback, rather than in training (Issenberg et al. 1999; Walsh et al. 2009).

Feedback and evaluation play important roles in simulation-based medical education and they affect learning through their format, content, and programming. To maximize learning, feedback and evaluation should be appropriately integrated into simulation-based training. However, the best integration of them should be investigated in the future.

Several design limitation of this study should be concerned. Better outcome in E&F group could also be partially attributed to that they got familiar with how the evaluation would have been conducted through pre-training evaluation and preparation and familiarity might form to adapt to the post-training evaluation protocol. We could have exposed the control group to pre-training evaluation protocol without group feedback to investigate merely the impact of pre-training group feedback on students’ BLS skill acquisition. But we should not ignore the fact that pre-training group feedback could not be conducted without evaluation. So, our study was to investigate the combined impact of these two features of simulation education, evaluation and feedback. Another limitation of this study is that the two instructors in both groups could not be blinded because duration of BLS training was different between the two groups. Therefore, personal bias could not be avoided. Time spent for pre-training evaluation was another bias factor when we identify better outcome in the E&F group. In the E&F group, higher acute rates were not observed in all of the five discrete BLS skills (correct compression site was comparable between the two groups). This result showed although the 23-item checklist was good tool to assess trainees’ proficiency of the whole BLS algorithms, it is not always consistent to reflect competency of each discrete BLS skill. That was why we added accurate rate comparison of five discrete BLS skills that were emphasized by AHA because those discrete BLS skills could influence the outcomes of BLS. Furthermore, the potency and reliability of the checklist had not been tested in this study and should be identified in the future.

Conclusions

In undergraduate medical students without previous BLS training, pre-training evaluation and feedback improve their performance in immediately followed BLS training. A future study with longer follow-up periods to estimate the effect on long-term retention of learning outcome is needed.

Acknowledgments

This study was funded by a Medical Education Research Grant from Medical Education Committee of Chinese Medical Association and Medical Education Association of High Education Society of China in 2010. Grant number: 2010-15-06. The authors thank the medical students who participated in the study, the support from Clinical Skills Training Center, West China Hospital of Sichuan University. The authors gratefully acknowledge the assistance of Dr Douglas Hester from Vanderbilt Medical Center, TN, USA, in critically reviewing this article.

Declaration of interest: The authors report that there is no commercial or non-commercial affiliation that are or may be perceived to be a conflict of interest with the work. The authors alone are responsible for the content and writing of this article.

Appendix 1

Questionnaire of BLS

Table

Name	Sex	Age	Grade
Telephone			E-mail
1 How many times have you received lecture of BLS before?		□ One	□ Two		□ Three or above
2 How many times have you received training of BLS before?		□ One	□ Two		□ Three or above
3 Are you satisfied with this training program?		A Yes, I like it.	B It is ok.		C No.
Please give the detail reason if you select C!
4 Do you think this training is instructive?		A Yes, excellent	B Yes, it is.		C No.
5 Which of the following methods of training do you prefer?
A PPT presentation	B DVD demonstration	C Manikin demonstration			D Combination of above three
6 Do you like to receive our questionnaire again 3 months later?
A Yes, I do	B No, I don’t
7. Other suggestions and advice for our training:

Test of BLS

• 1 A 45-year-old man fell unconscious when he did exercise in a fitness center, where should you check his pulse to identify whether he had cardiac arrest?

• A To check the pulse of the radial artery.

• B To check the pulse of the brachial artery.

• C To check the pulse of the carotid artery.

• D To check the heartbeat.

• 2 According to the 2005 AHA guideline of BLS, please describe the right sequence of BLS________________________________.

• A Call for help.

• B Press the nasolabial fold.

• C Wait for the arrival of the rescuers.

• D Perform the chest compression at once.

• E Perform the mouth-to-mouth breath at once.

• F Check the pulse.

• G Check the breath.

• H Check the conscious.

• 3 What is the correct frequency of the chest compression?

• A Faster and better.

• B 70–120 bpm for the adult.

• C 100 bpm both for adult and children.

• D 100 bpm for adult and 150 bpm for children.

• 4 The best frequency of artificial ventilation is ().

• A Faster and better.

• B Three ventilations per minute for adult.

• C Two artificial ventilations first both for adult and children, and then followed by other treatment.

• D 10–15 ventilations per minute for adult and 20 ventilations per minute for children.

• 5 How to check the patient's breath?

• A Put your finger close to the patient's nostril and feel the air-flow of breathing.

• B Watch the movement of the chest wall.

• C Put your ear close to the patient's nose, watch the chest movement, listen to breathing sound, feel the air flow of breathing.

• D Head tilt and chin lift to open the airway.

• 6 How to check the patient's consciousness?

• A To tap the patient's face.

• B To tap the patient's forehead.

• C To tap the patient's chest.

• D To tap the patient's shoulder.

• 7 How long should breath check last?________

• 8 How long should pulse check last?________

• 9 How to open the airway for a patient with suspected neck injury?

• A Head tilt and chin lift.

• B To try to thrust his jaw and keep his neck still. If it is impossible, to perform head tilt and chin lift.

• C To use your finger to clean foreign body in his mouth and pull his tongue out of his mouth.

• D Since neck injury was suspected, leave him alone.

• 10 What's the correct ratio of chest compression to artificial ventilation during BLS? ______:_____

• 11 Where should you put your hands on to perform the chest compression?________

• A The upper half of the sternum

• B The left chest, nipple level and above the heart

• C The bottom of the sternum, above the xiphoid

• D The cross-site between the sternum and the nipple line

• 12 Which of following descriptions are correct for BLS?

• A A hard plate should be placed under the patient's body

• B A frequency of 100 bpm for chest compression

• C Hard push

• D Make sure the chest wall was fully recoiled

• E Make sure your palm was away from the patient's chest during ventilation

• Note: bpm, beat per minute.

Appendix 2

Item checklist for basic life support skill evaluation

Table

01	Correct consciousness judgment (tap patient's shoulder)
02	Call for help and AED
03	Airway opening (head tilt and chin lift)
04	Correct breathing checking (looking, listening, and feeling)
05	Breathing checking time (5–10 s)
06	Keep airway opening (head tilt and chin lift)
07	Correct mask ventilation (“C” shape for the thumb and index and “E” shape for the other three fingers)
08	Each ventilation lasts 1 s
09	Effective ventilation (both visible chest wall movement in Resusci® Anne Basic and SkillGuide™ show green light for ventilation)
10	Correct pulse checking site (ipsilateral carotid artery)
11	Pulse checking time (5–10 s)
12	Correct chest compression site (cross-site between nipple line and middle-line of sternum)
13	Shoulders right over the hands
14	Straight elbows
15	Fingers extended and inter-sectioned
16	Fingers off the chest
17	Count aloud
18	Chest wall fully recoil
19	The hands should never be away from the chest wall during compression
20	Correct compression depth (SkillGuide™ shows green light for compression)
21	Correct compression rate (30 compressions was completed between 16 and 20 s, which was approximately 90–110 compressions per minute)
22	Ratio of compression / ventilation (30:2)
23	CPR-free interval less than 15 s (between each circle including pulse re-checking)

Notes: AED, auto electric defibrillator; CPR, cardiopulmonary resuscitation.

Resusci® Anne Basic and SkillGuide™, Laerdal Medical Corporation, Stavanger, Norway.