Evaluation of teachers training workshop at Kirkpatrick level 1 using retro–pre questionnaire

Abstract

Background

Teacher training focusing on teaching learning methodologies, assessment tools, and approaches that motivate the teachers and enhance their confidence is one of the fundamental programs of faculty development. The objective of this study was to assess the self-reported perceived confidence of faculty members after participating in a basic teachers training workshop.

Methodology

The retrospective post-then-pre design questionnaire was used for evaluating the faculty members’ self-reported perceived confidence at Kirkpatrick level 1 (evaluation of reaction) after completing basic teachers training workshops. The self-reported perceived confidence on 30 statements were analyzed by 3 categories (not confident, confident, and highly confident) on a Likert scale.

Results

Out of 60 participants (18 in the 3rd and 21 each in the 4th and 5th teachers training workshops), 58 (96.67%) responded to all statements. The mean age of the participants was 34.14±5.15 years; 70.7% were male and 29.3% female. Overall, the self-reported perceived confidence level of the participants of the 3rd (pre-training median 57, post-training median 70, p<0.001), 4th (pre-training median 51, post-training median 67, p<0.001) and 5th (pre-training median 51, post-training median 68, p<0.001) training workshops was significantly increased after training. There was a noticeable change in the level of confidence of the participants after training. The observed percentage change in self-reported perceived confidence was 29.63% in those who received no training prior to joining college (n=51, 87.9%); this was 2 times more than those who received some sort of training prior to teachers training (n=7, 12.1%).

Conclusion

Overall, the self-reported perceived confidence level of the participants was significantly increased after teachers training workshop.

Video abstract

Point your SmartPhone at the code above. If you have a QR code reader the video abstract will appear. Or use:

http://youtu.be/HBqzVzOFZY8

Keywords:

• assessment
• confidence
• Kirkpatrick level 1
• Nepal
• retro–pre
• teachers’ training

Introduction

Medical education comprises 3 key components: a curriculum, an educational environment and the teachers or faculty members. Teachers not only influence the teaching and learning process but also play an essential role in shaping the other 2 components: a curriculum and an educational environment. To improve and maintain the quality of medical education, faculty development programs are indispensable. Teachers training focusing on teaching learning methodologies, assessment tools, and approaches that motivate the teachers and enhance their confidence, and it is one of the fundamental programs of faculty development. This enriches the capacity of faculty in teaching and learning and assessing students’ performance.^1–5

The evaluation of teachers training program documents the enhancement of knowledge and teaching skills of the trained teachers.⁶ Retrospective post-then-pre questionnaire design is one of the instruments used to evaluate the efficacy of training program. It assesses the learners’ self-reported changes in knowledge, awareness, skills, confidence, attitudes, or behaviors. It is used to avoid the sensitivity of pretest and reduce the speculation.^7,8

Health Professionals Education and Research Center, Chitwan Medical College (CMC), Bharatpur, Nepal, has been organizing teachers training workshop since September 2014 in order to strengthen the capacity and competence of faculty members in teaching learning and assessment methods and to boost their level of confidence in this aspect. The main objective of this study was to assess the self-reported perceived confidence of faculty members after participating in basic teachers training workshop at Kirkpatrick level 1 using retro–pre questionnaire, but the perception at level 2a and, to some extent, knowledge/skills at level 2b were also assessed. The rationale of the study was as follows: 1) participants’ reactions, perception, and self-confidence can usually be measured with retro–pre questionnaire immediately after the training program, and 2) it is believed that the positive self-reported perceived confidence may motivate teachers and bring positive behavioral changes among them for teaching learning, educating the students, and assessing their performance.^9,10

Methodology

The retrospective post-then-pre design questionnaire was used for assessing faculty members’ self-reported perceived confidence after completing basic teachers training workshop at Kirkpatrick level 1 (evaluation of reaction). In addition, the perception at level 2a and, to some extent, knowledge/skills at level 2b were also assessed with the same questionnaire. There are 4 levels of Kirkpatrick’s model of evaluation – level 1: evaluation of reaction; level 2: evaluation of learning (level 2a: attitudes/perceptions and level 2b: knowledge/skills); level 3: evaluation of change in behavior; and level 4: overall impact of training (level 4a: organizational practice, level 4b: student benefit and level 4c: patient benefit).^9,11

After piloting the retro–pre questionnaire in the 2nd teachers training workshop, it was subsequently used for the assessment during the 3rd, 4th and 5th teachers training workshop. Each statement was assessed using a 4-point Likert scale: not confident, somehow confident, very confident and highly confident for both pre and post responses.

Written informed consent was taken from the participants.

Each teachers training workshop was of 6 days duration. The 3rd, 4th and 5th teachers training workshops were held at Health Professionals Education and Research Center, CMC, Bharatpur, Nepal, during July 26–31, 2015; February 27–March 3, 2016; and September 5–10, 2016, respectively. Altogether, 60 (18 in the 3rd and 21 each in the 4th and 5th workshops) faculty members participated in the teachers training workshop from clinical sciences, basic sciences, public health, and nursing departments of CMC. The national and regional resource persons facilitated the training besides local resource facilitators.

The main sessions conducted were on educational objectives, curriculum and role of faculty member, principle of adult learning, behavioral objectives, feedback skills, communication skills, facilitation skills, interactive teaching/learning, small group discussion, role play, clinical skills teaching/learning, community-based learning, ethics for professional educators, microteaching, lesson planning, audiovisual aids, problem-based learning, and assessment tools. Most of the sessions contained a tutorial on the subject with brainstorming, group work in small groups (3 small groups with 6–7 participants in each) and presentation of group work in a large group with discussion.

The collected data were checked for completeness, accuracy and consistency. The collected data were coded and entered in Epi Data 3.1 (EpiData Association, Odense, Denmark) and exported to IBMS SPSS version 20 (IBM Corporation, Armonk, NY, USA) for analysis. Data were categorized as 0 for not confident, 1 for somehow confident, 2 for very confident and 3 for highly confident. Later, the “somehow confident” and “very confident” categories were merged into the “confident” category. Altogether, 30 statements were used, and the maximum possible score was 90 for each participant. The data were analyzed by frequency, percentage, mean (standard deviation) for continuous variables and median (interquartile range) for ordinal variables, and p-values were calculated for significance. p-values were computed using Wilcoxon signed-rank test for the pre and post scores of self-perceived confidence of the participants, while Mann–Whitney U-test was used for computing p-values for comparison between the 2 groups.

The details of the groups were as follows:

Group I – received some sort of training on teaching learning before joining CMC.

Group II – received no training on teaching learning before joining CMC.

Results

Out of 60 participants (18 in the 3rd and 21 each in the 4th and 5th workshops), 58 responded to all statements included in the retro–pre questionnaire. The response rate was 96.67%. Males accounted for 70.7% of the participants, while females were 29.3%. The mean age of the participants was 34.14±5.15 (min, 25 years and max, 54 years). More than half of the participants (51.7%) were engaged in teaching learning prior to joining the CMC, while the remaining (48.3%) joined CMC immediately after receiving a postgraduate qualification. Seven (12.1%) participants received some sort of training on teaching learning before joining CMC.

The self-reported perceived confidence on 30 statements/items were analyzed for 3 categories (not confident, confident and highly confident) using a Likert scale. There was a marked difference found in the level of confidence of the participants after training (Table 1).

Table 1 Self-reported perception and confidence of the faculty members before and after teacher training workshop on various sessions conducted during training

S. No	Statement	Not confident		Confident		Highly confident
		Before	After	Before	After	Before	After
1.	Curriculum and role of faculty	22.4	0	74.1	81.0	3.4	19.0
2.	Giving feedback (feedback skills)	10.3	0	86.2	70.7	3.4	29.3
3.	Communication skills	8.6	0	89.7	69.0	1.7	31.0
4.	Facilitation skills	19.0	1.7	77.6	67.2	3.4	31.0
5.	Principles of adult learning	36.2	1.7	63.8	79.3	0	19.0
6.	Writing behavioral objectives	39.7	0	56.9	70.7	3.4	29.3
7.	Conducting small group discussions	22.4	1.7	70.7	60.3	6.9	37.9
8.	Conducting role plays	31.0	1.7	69.0	65.5	0	32.7
9.	Ethics of teaching/learning	17.2	1.7	81.0	72.4	1.7	25.9
10.	Understanding microteaching	32.8	1.7	65.5	69.0	29.3	29.3
11.	Developing a lesson plan	32.7	0	60.3	55.2	6.9	44.8
12.	Using audio–visual aids	8.6	0	79.3	41.4	12.1	58.6
13.	Designing problem-based learning	43.1	3.4	55.2	77.6	1.7	18.9
14.	Conducting problem-based learning sessions	29.3	0	69.0	77.6	1.7	22.4
15.	Conducting assessments	12.1	0	77.6	65.5	10.3	34.5
16.	Designing multiple choice questions	18.9	0	69.0	60.3	12.1	39.7
17.	Conducting multiple choice question examination	13.8	1.7	70.7	56.9	15.5	41.4
18.	Conducting modified essay question examinations	46.6	3.4	50.0	74.1	3.4	22.4
19.	Designing modified essay question	41.4	1.7	55.2	79.3	3.4	19.0
20.	Social accountability of medical professionals	17.2	1.7	77.6	75.9	5.2	22.4
21.	Designing OSCE	36.2	3.4	60.3	65.5	3.4	31.0
22.	Conducting OSCE	32.8	3.4	65.5	62.1	1.7	34.5
23.	Designing OSPE	31.0	0	63.8	70.7	5.2	29.3
24.	Conducting OSPE	39.3	0	67.2	62.1	3.4	37.9
25.	Conducting oral examination (viva)	10.3	1.7	81.0	55.2	8.6	43.1
26.	Role of a teacher	8.6	5.2	82.8	67.2	8.6	27.6
27.	Community-based learning	37.9	3.4	60.3	86.2	1.7	10.3
28.	Delivering interactive lectures	20.7	0	77.6	74.1	1.7	25.9
29.	Educational objectives	27.6	1.7	67.2	77.6	5.2	20.7
30.	Transmitting/sharing knowledge	12.1	0	84.5	72.4	3.4	27.6

Abbreviations: OSCE, objective structured clinical examination; OSPE, objective structured practical examination.

Overall, the self-reported perceived confidence level of the participants was significantly increased after teachers training workshop (Table 2).

Table 2 Overall self-reported confidence level of the participants before and after teachers training workshop

Training	n	Pre	Post	p-value
		Median (interquartile range)	Median (interquartile range)
Third	18	57 (52.37–62)	70.5 (60–70.5)	<0.001
Fourth	19	51 (48–60)	67 (62–72)	<0.001
Fifth	21	51 (46.5–58)	68 (60.5–75)	<0.001

Note: p-value is computed using Wilcoxon signed-rank test.

Participants (n=7, 12.1%) who received some sort of training on teaching learning before joining CMC were comparably more confident prior to teachers training than those who received no training on teaching learning before joining CMC (n=51, 87.9%) (p-value <0.001). Although the level of confidence after teachers training was comparably increased more in those participants who received no training on teaching learning before joining CMC, this was not statistically significant (p-value =0.141). The observed median percentage change in self-reported perceived confidence was 29.63% after teachers training in those who received no training before (n=51, 87.9%); this was 2 times more than those who received some sort of training prior to teachers training (n=7, 12.1%), and the result is statistically significant (p-value 0.018) (Table 3).

Table 3 Comparison of self-reported perceived confidence level of the participants who received no training on teaching learning compared to those who received some sort of training (before and after teachers training workshop)

Received some sort of training on teaching learning	n	Pre	Post	Percentage change
		Median (interquartile range)	Median (interquartile range)	Median (inter quartilerange)
No	51	52 (48–58)	66 (60–74)	29.63 (18.46–42.73)
Yes	7	65 (60–68)	71 (70–74)	10.96 (7.69–16.39)
p-value		<0.001	0.141	0.018

Note: P-value is computed using Mann–Whitney U-test.

Discussion

Faculty members represent one of the key resources at any medical college. The academic success of a medical college depends upon the performance of faculty members.¹² Teachers training may improve quality, professionalism, and effectiveness of teaching and improve the self-confidence of the teacher, thus enhancing their performance.¹³ Our study assessed the self-reported perceived confidence of faculty members after participation in a basic teachers training workshop using retro–pre feedback questionnaire at Kirkpatrick level 1 (evaluation of reaction) besides the perception at level 2a and, to some extent, knowledge/skills at level 2b. It is anticipated that the positive self-reported perceived confidence may motivate teachers and bring positive behavioral changes among them for teaching learning, educating the students, and assessing their performance.¹⁰

Our study revealed that after teachers training, the highest percentage of participants perceived themselves as being highly confident in using audio–visual aids (58.6%), developing a lesson plan (44.8%) and conducting oral examination – viva (43.1%), whereas the least percentage of the participants perceived themselves as being highly confident in community-based learning(10.3%), designing problem-based learning (18.9%), knowledge on curriculum and role of faculty (19%), principles of adult learning (19%) and designing modified essay questions for examination (19%). The low score could possibly be due to less time allocated for these sessions.

Overall, the self-reported perceived confidence level of the participants of the 3rd (pre-training median 57, post-training median 70, p<0.001), 4th (pre-training median 51, post-training median 67, p<0.001), and 5th (pre-training median 51, post-training median 68, p<0.001) training workshops was significantly increased after training.

The retro–pre questionnaire was used to assess the learners’ self-reported changes by Rajalakshmi et al¹⁴ at Kirkpatrick level 1 (evaluation of reaction). They evaluated participants’ reaction (self-perceived) at the end of workshop on communication skills for health care providers and reported significant self-perceived improved changes in communication skills of the participants.¹⁴ Masood and Usmani¹¹ evaluated medical teacher training program through Kirkpatrick’s model at all 4 levels. Participants in their training perceived that their learning considerably improved after training at Kirkpatrick level 1 (evaluation of reaction).^9,11

Our study also showed a comparably increased level of self-reported perceived confidence after teachers training workshop among the participants who received no training on teaching learning prior to the training workshop (percentage change: 29.63) compared to those who had received some sort of training (percentage change: 10.96). Statistically, this difference is significant.

The findings may not be generalized as this study was conducted in 1 institution. This type of research must be conducted in other institutions organizing similar faculty development programs.

Conclusion

Overall, the self-reported perceived confidence level of the participants was significantly increased after teachers training workshop. A significant change in the level of self-reported perceived confidence was noticed after teachers training workshop among the participants who received no training on teaching learning prior to the teachers training workshop. Further research is required to explore the behavioral changes among the teachers and their teaching performance after teachers training program at Kirkpatrick level 3 and in terms of organizational practice (Kirkpatrick level 4a), student benefit (Kirkpatrick level 4b) and patient benefit (Kirkpatrick level 4c).

Disclosure

The authors report no conflicts of interest in this work.