Introducing computer-assisted training sessions in the clinical skills lab at the Faculty of Medicine, Suez Canal University

Abstract

Background: The Faculty of Medicine, Suez Canal University clinical skills lab was established in 1981 as the first skills lab in Egypt to cope with innovation in medical education adopted since school inauguration in 1978. Students are trained using their peers or models. Training is done weekly, guided by checklists tested for validity and reliability and updated regularly. Students receive immediate feedback on their performance. Recently, the number of students has increased, leading to challenges in providing adequate supervision and training experiences. A project to design and implement a computer-assisted training (CAT) system seemed to be a plausible solution.

Aims: To assess the quality of a newly developed CAT product, faculty and students' satisfaction with it, and its impact on the learning process.

Methods: The project involved preparation of multimedia video-films with a web interface for links of different scientific materials. The project was implemented on second year students. A quality check was done to assess the product's scientific content, and technical quality using questionnaires filled by 84 faculty members (139 filled forms) and 175 students (924 filled forms). For assessment of impact, results of examinations after project implementation were compared with results of 2nd year students of previous 3 years.

Results: More faculty (96.3%) were satisfied with the product and considered its quality good to excellent, compared to 93.9% of students, p < 0.001. Most faculty (76.2%) have agreed on its suitability for self-learning, while most students considered the product would be suitable after modification. The percentage of students' failures was lower after project implementation, compared to previous 3 years, p < 0.05.

Conclusion: CAT materials developed for training of second year students in skills lab proved to be of good scientific content and quality, and suitable for self-learning. Their use was associated with lower failure rates among students. A randomized trial is recommended to ascertain the effectiveness of its application.

Background

Training of medical students in basic clinical skills is an essential component in medical education. However, the limited learning opportunities during clerkships for medical students make medical schools unable to rely on clerkship experiences alone to offer students sufficient basic clinical skills training (Remmen et al. 1999). This challenging situation has forced many medical schools throughout the world to establish skills laboratories to supplement skills training in their clerkships (Bradley & Bligh 1999; Da Costa et al. 2001). Skills laboratories are used to ease the learning of clinical skills (Ledingham & Harden 1998) and ensure that medical students have the required clinical competencies (Du Boulay & Medway 1999).

The Faculty of Medicine, Suez Canal University (FOM/SCU) was established in 1978, not just as an addition to other schools of medicine in Egypt, but was an initiative to innovation. The main concern of the founders of FOM/SCU, late professors Zohair Nooman and Esmat Ezzat, was to develop a curriculum responding to community needs (primary health care approach), bio-psychosocial paradigm, as well as to the explosion of information and principles of learning. It was also guided by the recommendations for overcoming the shortcomings of the traditional curricula spelled out in the symposium on medical education held at Fayoum, Egypt at the beginning of 1978, and attended by the Egyptian medical schools, Ministry of Health (MOH), and the Egyptian medical syndicate.

Table

Practice points
•Computer assisted learning (CAL) would help schools with high student: faculty ratios in clinical training.
•Students’ involvement in the process of production and evaluation of training materials is of great importance.
•Locally produced CAL materials might be better than just purchasing ready-made materials that may not suit the learning environment.
•Faculty members should be encouraged to participate in the production of their CAL materials.
•The application of trials or intervention studies in medical education research is urgently needed, but the limitations related to bylaws and regulations need to be solved first.

The Faculty of Medicine, Suez Canal University clinical skills lab was established in 1981 as the first skills lab in Egypt to cope with innovation in medical education adopted since school inauguration in 1978. The lab was aimed at standardizing training and evaluation of basic clinical skills, and promoting humanistic attitudes in a student's approach to patients. Training in the lab starts from year 1, and is integrated into the curriculum to serve the school's community-based education and problem-based learning strategies.

Students are trained using their peers (males examine males and females examine females) or models by junior staff members who have had prior training by specialized senior faculty. Training is done weekly, guided by checklists tested for validity and reliability and updated regularly. Students receive immediate feedback on their performance. Other learning materials used are handouts, illustrative charts, models for surface anatomy, video-films, and CDs.

Periodic and final evaluation is done for performance and background knowledge. Periodic evaluation is done every six weeks. Each student performs the skill on his/her colleague or on the model in front of the observers who use standard checklists. The observers are blind to the weight of each step. There are four alternatives for marking each step in the checklist (done satisfactorily, done unsatisfactorily, not done and not applicable). After each examination, the supervisors of the lab grade the checklists according to specific predetermined weight for each item and provide students with feedback after every examination. Students should prove mastery of all skills; otherwise they would repeat the examination until reaching the mastery level of performance. A final evaluation is done at the end of the year. The scoring of the theoretical background (written test) is done by the subject area experts according to an approved key answer sheet.

During recent years, the number of students has steadily increased, which led to logistic difficulties in skills lab training. The commonly reported problems are inability to follow a demonstration, lack of time for each student to practice, and dearth of the chance of getting feedback from instructors. Moreover, the specialists who train the instructors are not able to cope with the increasing hours of work in skills lab.

Thus, we thought of introducing a new learning method that overcomes these problems. Computer-assisted training (CAT) sessions seemed to be a plausible solution. We succeeded in implementing this solution through a project funded by the Higher Education Enhancement Projects Funds (HEEPF). The project was started in 2005, and applied using second year medical students. It aimed at production of standardized audio-video films for different skills, provided with computer-generated multimedia programs that are self-explanatory, interactive, and readily available at any time. We hypothesized that this instructional method would create an educational setting that fulfills highly standardized criteria.

Aims

The aim of this study was to assess the quality of the newly developed CAT product, the faculty and students' satisfaction with it, and its impact on the learning process, which would lead to implementing it on all classes.

Methods

The funding for the project, less than US$90.000, was mainly used in supporting the already existing equipment and infrastructure of the skills lab besides producing the video films. The funding agent (HEEPF) projects details can be found on their website (http://www.heepf.edu.eg). The project had three phases: preparation, implementation, and evaluation. The preparation phase involved preparation of video films. The basic principles of clinical skills teaching (based on taxonomy of psychomotor domain) were taken into consideration. These included conceptualization, visualization, verbalization, practice, correction and reinforcement, skill mastery, and autonomy (George & Doto 2001).

The process started with revision of the already used checklists in the skills lab (22 checklists) by senior experts in each pertinent specialty. Then, the project team invited senior experts for participation in demonstrating clinical skills. Their role in video capturing was clarified, and recommendations required to follow during recording were introduced to them. They were also asked to determine the models, charts, equipment and volunteers required for accomplishing the audiovisual aids. Timetable for skills recording was set.

A suitable room in the new faculty building was prepared, with furniture for video capturing. A server computer with a local network connected to 16 stations was established. Models required for practicing skills were prepared. The lecture hall was prepared and provided with a multimedia projector, and a computer for skills demonstration. Senior experts reviewed checklists before capturing to ensure they were followed. Supplies required for each skill were checked and prepared. Communication was made with volunteers to ensure their cooperation and to follow experts’ recommendations.

After capturing, movies were revised for matching with checklist items, and for arrangement, animation or graphics, and sections needing deleting. Movies were edited according to reviewers’ observations and were prepared in another format for installing as links as an alternative to the usual format of whole films. A web interface for different links (whole movie, movie sections, anatomical landmarks, abnormal signs, surface anatomy, common errors and MCQ) was designed.

The lab coordinators received training on operating the system and how to solve technical problems during its running. Tutors were asked to run the whole movie, allow students to practice under their supervision using checklists, replay sections which appear difficult or unclear for them, and re-practice to acquire competence.

The finalized multimedia programs were implemented on second year students. They were instructed to use checklists while running movies, identify unclear or difficult items, practice on their peers (or on the model according to skill) under the tutor supervision, and navigate through links for acquisition of required scientific information. All students in year 2 used the new method for all the recorded skills. The students attended the lab for two hours weekly all the year through. Any absent students in any session were allowed to observe the video films and practice outside the regular schedule of the lab. Also the lab was open for any student who wanted to revise any film outside the schedule.

A quality check of each computerized skill was carried out to assess its scientific content, steps of application of skill, use of illustrations, as well as video and audio qualities. This was done using two questionnaires, one for faculty, and one for students. Questionnaires were developed by project team members and were reviewed by experts in education and clinical skills training (both medical and technical) for completeness and relevance. The questions were simple and direct, and the responses were on a 4-point Likert scales: “yes”, “to a great extent”, “to some extent’, and “no.” These were respectively scored from “3” to “zero.” Mean and standard deviations were calculated for each sub-item, for the total of each item, and the total evaluation of the product. Movies were evaluated by 84 faculty members from specialties related to skill (139 filled forms), and 175 students (924 filled forms). Any student in year 2 was eligible for inclusion in the study sample, with no exclusion criteria. Similarly, any subject matter expert involved in a related skill was eligible for inclusion; indeed, almost all of them were included. There were no refusals to participate. The questionnaires were anonymous and they were distributed at the end of each session, both to students and tutors (faculty), they were filled and collected on-site; so there were no non-responders.

For assessment of impact of use of computer-assisted training materials, lists of failing students and the number of failures each year were prepared from year 2002 through year 2006. These were compared with the results of examinations after project implementation. Also, external evaluators from other universities participated in students’ evaluation. These were faculty members from other medical schools who were acquainted with our educational program. The exam method and setting were the same as in all other previous years' exams to be able to evaluate only the method of teaching.

Statistical analysis: Data entry was done using a Microsoft Excel-2000 computer software package, while statistical analysis was done using Epi-Info 6.04 and SPSS 10.0 statistical software packages. Quality control was carried out at the stages of coding and data entry. Categorical variables were compared using the chi-square test. Pearson correlation analysis was used for assessment of the relation between external and internal evaluators’ marks. Statistical significance was considered at p-value < 0.05.

Although the authors are part of the project team, procedures were taken to alleviate any conflict of interests. These procedures included anonymous data collection from both students and faculty members, and carrying out of exams by both internal and external examiners not including the authors. Furthermore, as the project was funded by the funding agent (HEEPF), steps were monitored and quality control measures were checked by external evaluators nominated by the funding agent.

Results

There was a quality check of each of computerized skill to assess scientific content, sequence of steps of application, use of illustrations, and video and audio qualities. Concerning scientific content as regards relevance, completeness, up-to-date state, and clarity, Table 1 shows that the majority of faculty (93.5%) considered it very good or excellent, compared to 77.7% of students, p < 0.001. Also, none of the faculty, and only 2.7% of students viewed scientific content as poor. Similar findings were revealed as regards sequence of steps, where more faculty considered it very good or excellent, compared to students, 84.9% and 70.7%, respectively, p < 0.001. As for use of illustrations, the majority of both groups considered it very good or excellent, but with more faculty, p < 0.001. Meanwhile, 2.2% of the faculty, and 4.3% of students had the opinion that use of illustrations was poor. As regards technical quality (audio/video), the rates of dissatisfaction were higher, reaching 20.2% for audio quality among students, and 4.3% among faculty. Still, faculty opinions were statistically significantly more positive, compared to students, p < 0.001. Overall, the majority of the faculty (87.0%) considered that the quality of the product was very good or excellent, compared to 58.0% of students, p < 0.001.

Table 1.  Students' opinions about total quality of total skills

	Faculty (n = 139)		Students (n = 924)		X^2 test
	No.	%	No.	%	p-value
Scientific content
Poor	0	0.0	25	2.7
Good	9	6.5	181	19.6	46.77
Very good	67	48.2	536	58.0	<0.001*
Excellent	63	45.3	182	19.7
Steps of skill application
Poor	0	0.0	34	3.7
Good	21	15.1	236	25.6	84.65
Very good	49	35.3	504	54.5	<0.001*
Excellent	69	49.6	150	16.2
Use of illustrations
Poor	3	2.2	40	4.3
Good	18	12.9	272	29.4	68.51
Very good	54	38.8	460	49.8	<0.001*
Excellent	64	46.0	152	16.5
Technical quality (video)
Poor	3	2.2	101	11.0
Good	19	13.7	270	29.2	83.33
Very good	54	38.3	419	45.3	<0.001*
Excellent	63	45.3	134	14.5
Technical quality (audio)
Poor	6	4.3	187	20.2
Good	16	11.5	245	26.5	82.99
Very good	60	43.2	369	40.0	<0.001*
Excellent	57	41.0	123	13.3
Total opinion
Poor	3	2.2	37	4.0
Good	15	10.8	351	38.0	89.37
Very good	88	63.3	492	53.2	<0.001*
Excellent	33	23.7	44	4.8

*Statistically significant at p < 0.05.

Concerning overall satisfaction with the product, Table 2 points to a statistically significant difference between faculty and students, p < 0.001. Slightly less than one third of the faculty (30.4%) have rated it excellent, compared to only 14.0% of students. Conversely, about one third of students (34.2%) have rated it good, compared to only 13.0% of faculty members. Only 2.2% of the faculty, and 1.4% of students had poor satisfaction with product.

Table 2.  Students' satisfaction with suitability for use in students’ training

	Faculty (n = 139)		Students (n = 924)		X^2 test
	No.	%	No.	%	p-value
Overall satisfaction with product
Poor	3	2.2	13	1.4
Fair	2	1.4	44	4.8
Good	18	13.0	316	34.2	42.18
Very good	73	52.9	422	45.7	<0.001*
Excellent	42	30.4	129	14.0
Product suitable for students training
No	14	10.1	55	6.0
Yes after modification	19	13.7	574	62.1	116.95
Yes	106	76.2	295	31.9	<0.001*

*Statistically significant at p < 0.05.

The same table indicates a wider discrepancy between faculty and students as regards opinion about suitability of product for students’ training. While more than three-quarters of the faculty (76.2%) have agreed on suitability, the highest percentage of students had the opinion that the product was suitable after modification. Meanwhile, 10.1% of faculty members, and 6.0% of students have disagreed on product suitability.

When students were asked about intention to use the product in self-training, less than half (48.9%) have confirmed they will use it, and another third (34.9%) had the intention to frequently use it (Table 3). Conversely, 6.5% of the students had no intention for use.

Figure 1. Correlation between external and internal evaluators’ assessment of students performance in skills lab after use of computer-assisted training.

Table 3.  Students’ satisfaction with skills’ suitability for use in self training*

Will use this product in self training	Frequency	Percent
No	60	6.5
Rarely	89	9.7
Frequently	322	34.9
Yes	451	48.9

*2 missing.

As Table 4 illustrates, the number of failed students and failure times had a decreasing trend throughout analysis years. The rate dropped from 73/207 in year 2002/03 to 2/175 in year 2005/06, the year of implementing computer-assisted skills training. However, the percentage of failures was statistically significantly lower after implementation of the project, compared to previous 3 years, p < 0.05. Furthermore, external evaluators have participated in students’ assessment during this year (2005–06) to increase validity and reliability. As Figure 1 shows, there was a strong correlation between external and internal evaluators’ marks, pointing to consistency of results.

Table 4.  Comparison of students’ assessment in skills after use of computer-assisted skills training with previous years

	Previous years
	02/03	03/04	04/05	05/06
	(n = 207)	(n = 187)	(n = 168)	(n = 175)
No. of failure times:
1	73	17	7	2
2	0	5	1	0
3	0	3	0	0
Total students failed	73	25	8	2
Total times failure	73	36	9	2
X^2 test (comparing 05/06  with previous years)	69.98, p < 0.001*	31.55, p < 0.001*	4.90, p = 0.03*

(*)Statistically significant at p < 0.05.

Discussion

Skills training in a skills laboratory is an important step that guarantees that medical students acquire essential psychomotor skills in a safe protected environment, with the absence of anxiety or stress from direct exposure to real patients. It also has important ethical backgrounds. According to Nielson et al. (2003), the great majority of medical students found that training in a skills laboratory improved the outcome of the early clerkship.

Although medical students and residents are adult learners and although medical education should follow principles of adult learning, this does not usually happen. Schwenk (2004) has set forward some principles to enhance the adult learning process. This author has emphasized that adults usually want to apply what they learn soon after they learn it, they are interested in learning concepts and principles, and in solving problems rather than just learning facts. They like to participate actively in the learning process, and like to know how well they are doing. Computer Assisted Learning (CAL) in undergraduate medical education might be a good means to achieve these principles (Greenhalgh 2001). Moreover, according to JHPIEGO (2003), it has many advantages both to students and organization in terms of active participation, time saving, and cost-effectiveness.

The authors were mindful of these principles when constructing the computer-assisted skills lab in the present study. Active participation of students in the evaluation of video-films to be used in skills training was an important milestone in quality check of the product. In fact this proved to be very useful in improvement of both the content and quality of the films. According to study findings, concerning evaluation of scientific content of skills, it was shown that students had a more critical evaluation, compared to faculty. This difference might be explained by the basic concepts of adult learning, where the learner feels responsible for his/her own education, and thus looks for excellence. This was quite evident in their evaluation of the scientific content and steps of skill application, which were more valued by faculty, compared to students. Although the differences between faculty and students’ views might reflect a more realistic faculty opinion about what the second year student should know and be able to do, students’ opinions were actually used to improve the scientific content and background of skills, and also in rearranging some steps. Nevertheless, the majority of students and faculty have agreed on the high quality of the films regarding their scientific content and sequencing.

As regards technical quality of the video-films, students were less satisfied than faculty members with the audio and video quality, and the differences were statistically significant. This was also true for use of illustrations. The differences between students and faculty might be attributed to a generation gap. These students, currently in their late second decade of life, are far more familiar with computer, graphics, videogames, and Hi-Fidelity sounds and music. Their judgment about these technical issues is expected to be more critical, compared to faculty members who have started interaction with computer late in their life. This critical evaluation by students was very helpful in perfecting audio-video quality of films, and to reach to a level that responds to their demands.

Based on the foregoing, it was expected that faculty would be more satisfied with the product, and would consider it suitable for students’ training. This was exactly what was found, where the highest percentage of students had the opinion that the product was suitable after modification. The suggested modifications were actually done, based on students’ proposals, and faculty opinions about technical quality and content of the video-films.

As regards students’ intention to use the product in self-training, most of them have agreed that they would use it. Nonetheless, still 6.5% of the students frankly expressed their intention not to use it. This finding is in line with Greenhalgh (2001) who has reviewed the results of Computer Assisted Learning in Undergraduate Medical Education. Accordingly, it has been shown that some students made no use of their computers at all. This was attributed to technical problems, problems with staff being ill prepared for the change, and lack of academic organizational structure. The author added that initiatives to develop computer-based materials usually begin as distinct projects with management separate from existing structures and process of the university, which inevitably limits its benefits to the organization and limits its expansion and sustainability. In our case, the fact that the project is an integral part of the medical school educational system and the principal investigator is the head of the educational sector provides a guarantee for sustainability of the project.

The issue of effectiveness of computer-assisted training in medical education was a source of debate. According to JHPIEGO (2003), computer-assisted learning enhances instructional effectiveness. However, as regards participant achievement, studies could not show significant differences between achievements of individuals who learn through computer-assisted methods with those taught in instructor-led classes. This was noticed in terms of grades, test scores, retention and job performance, regardless of content, participant educational level or media used.

In the present study, the percentage of failures of students to pass the skills lab exam was statistically significantly lower after project implementation, compared to the previous three years. The results were further supported by demonstration of a strong correlation between external and internal evaluators’ marks, which adds to reliability of the assessment tools. Although this might imply effectiveness of the project in improving students’ performance, it cannot be totally attributed to computer-assisted learning. A randomized trial is needed to demonstrate this, since comparison with results of previous years might be confounded with several factors related to differences in students, staff, and educational environment. Although the confounding effect could be adjusted through multivariate analysis, this might need information that was not available. Nevertheless, a prospective controlled intervention trial would provide better evidence of effectiveness.

Conclusions

We conclude that the Computer-Assisted Learning materials developed proved to be of good scientific content and quality and suitable for self-learning, as judged by students and faculty members. Their opinions were of considerable importance in improving it. The use of this technology was associated with lower failure rates among students. However, a randomized trial is needed to ascertain the effectiveness of its application. Moreover, long-term follow-up is needed to assess students’ utilization, and the relation between frequency of use and students’ performance in the examination and clinical settings, and later in clerkship training.

Additional information

Notes on contributors

Somaya Hosny

SOMAYA HOSNY, PhD, MHPE, is Vice Dean of Education, Director of the Educational Development Center, Chairman of Histology & Genetics Department, and PI of the CAT project.

Adel M. Mishriky

PROFESSOR ADEL M. MISHRIKY, is ex-chairman of Community, Environmental, and Occupational Medicine.

Mirella Youssef

MIRELLA YOUSSEF, is lecturer, Community Medicine, co-ordinator of Skills Lab, and Co-PI of the CAT project