Helping teachers to embed e-learning materials into undergraduate pharmacology courses

Abstract

A project with the goal of testing a strategy to facilitate teacher’s use of existing e-learning resources in pharmacology is described. Experienced pharmacology teachers from a number of UK universities were recruited to develop editable teaching and learning materials (Teaching and Learning Resource Packs: TLRPs) designed to better enable teachers to embed CAL resources into their courses. Qualitative (questionnaires, structured interviews, diaries) and quantitative (knowledge tests) evaluation tools were developed and administered in a 3-stage process i.e. to students and staff in: the universities who developed the TLRPs (stage1); other universities associated with the project (stage 2); universities not associated with the project (stage 3). The evaluation process sought to collect data which would: inform the process of TLRP development; explore attitudes and perceptions of students and teachers to the use of CAL/TLRP resources; assess the effectiveness of the teaching session. Preliminary evaluation data are presented and discussed which suggest that the project outputs (exemplar TLRPs) have made it easier (saved time in producing support materials) for pharmacology teachers to successfully integrate existing CAL resources into courses thus making the original investment in those resources more cost-effective. The data also suggest that this mode of teaching has delivered student learning at an appropriate level and proved to be acceptable to both staff and students.

Keywords:

• e-learning
• computer-assisted learning (CAL)
• pharmacology teaching
• undergraduate

Introduction

The use of e-learning resources to support pharmacology teaching is now widespread. The pharmacology discipline has benefited significantly over the last decade from major UK government-funded initiatives notably the Teaching & Learning Technology Programme (TLTP) - pharma-CAL-ogy (1994–97), and the Pharmacy Consortium (PC)CAL (1992–1995). The products of both projects were predominantly interactive computer-assisted learning (CAL) materials based around specific elements of a course. Many universities, particularly in the UK, have acquired at least some of these materials and made them available to students, via local area networks, as support resources. However, there seems to be little evidence of them being integrated into courses despite the results from a number of studies, in which the usefulness of such programs has been evaluated, and which suggest that using such resources to replace traditional modes of course delivery can be effective (e.g. Dewhurst et al., 1994; Fawver et al., 1995; Guy & Frisby, 1992; Hughes, 2001; Kinzie et al., 1993; Leathard & Dewhurst, 1995; Johnson et al., 1997; Dewhurst & Williams, 1998; Dewhurst et al., 2000).

In 1998 a survey of pharmacology teachers (Markham et al., 1998) showed that 80% or more are aware of technology based teaching and learning methods for use in pharmacology but only 23% are actively using such resources. The same study reported that the main constraints impeding their use were: the lack of recognition and status given to innovative teachers; and, the pressure of other commitments preventing the large initial investment of time required to develop appropriate staff skills and to embed technology based teaching and learning in a course. Another study has demonstrated clearly that unless CAL resources are integrated and assessed then the evidence is that students will not make use of them (Hughes, 2000).

The TLTP recognised this as a problem across many disciplines and in 1998 initiated a further round of funding (TLTP3: £10.5 million) with a focus on developing strategies to encourage embedding of existing CAL materials into courses. The pharmacology discipline was represented by a consortium project led by the University of Leeds - ‘Implementing technology-based teaching and learning in pharmacology’. The goal of this project was to develop a range of flexible, wrap-around support materials (Teaching and Learning Resource Packs (TLRPs)) designed to help teachers to integrate existing CAL resources developed as a result of TLTP phase 1 and 2 funding into their courses. These might be used ‘off-the-shelf’, be modified by teachers to suit local needs or act as exemplars to assist teachers in developing their own support materials.

This paper will focus on reporting the strategy and tools adopted for both formative and summative evaluation of the project’s outputs and on describing some of the initial evaluation data.

TLRP Development and Evaluation Methodology

TLRP Development

The development of the materials required to support the use of e-learning resources is time consuming and requires certain skills. These materials might include, for example, the schedules setting out the learning goals the student is expected to achieve; the tasks the student is expected to accomplish using the software; schedules assessing the skills and learning gained. It is these tasks, activities and assessments which form a TLRP. Although there is considerable diversity in the range of activities which make up different TLRPs, a typical example might include: a set of multiple choice questions and extended matching sets with answers; a task requiring the setting of multiple choice questions; help in setting tasks requiring the production of a poster or delivering a talk i.e. exemplar poster titles and instructions on how to prepare a poster; creating web-pages; progressing through a work-book; problem-based learning exercises, designed to be suitable for different types of student at different levels; and assessments.

Table 1 The project resulted in the development of 12 TLRPs each designed to support either a single (or in some instances a small group) of existing CAL programs developed in previous TLTP-funded projects. The developed TLRPs were deployed with the CAL program in the university departments in which they were developed (stage 1) and then in other project-consortium university departments (stage 2). Their use by both students and teachers was evaluated via pre- and post-TLRP use questionnaires and structured interviews. Stage 1 questionnaire data has been collected from twelve of the twenty TLRP development teams (student: pre-TLRP n=745, post-TLRP n=620; staff: pre-TLRP n=4, post-TLRP n=6). The number of completed staff questionnaires is extremely disappointing and probably too small to enable any useful conclusions to be drawn. Currently, only 4 TLRPs have been evaluated in stage 2.

The project delivered 20 TLRPs over three years (Table 1) each designed by a group of pharmacologists to help the teacher to integrate software package(s) into a course/module without a large expenditure of time. They thus contain a synthesis of ideas from a number of staff and are extremely diverse in their approaches. The brief was to develop a range of support materials suitable for different student groups — pharmacology, biological science, medicine, dentistry and nursing. All materials were made available in MS Office formats (text-based material in Word, numeric material in Excel, presentational material in PowerPoint), so that it is a relatively simple matter for a teacher to put together a learning resource by selecting ready-made exercises, or by tailoring existing ones to their own needs and adding new ones.

Evaluation

Evaluation of the project was managed by one of the project consortium’s member universities (Edinburgh), which was not a TLRP development site and therefore was independent of the product development side of the project. An evaluation strategy was developed which was felt to be appropriate for a multi-centre project (20 development teams consisting of academic teachers from several institutions). The key stakeholders in the project, i.e. those who are best able to effect change, were pharmacology teachers who are largely unfamiliar with qualitative research methods and sceptical of the value of data they produced.

The evaluation strategy developed, with advice from external advisors (Tavistock Institute) to the TLTP, focussed on the quality of the product (TLRP) development, its effectiveness in enhancing the use of CAL programs in pharmacology teaching and the process of TLRP. Both staff and student users were surveyed (questionnaires, structured interviews) to elicit their perceptions of the effectiveness and usability of a CAL/TLRP compared to traditional methods of teaching which may be replaced by this approach and the usefulness of the teaching it delivered. Knowledge tests were used to measure the learning experienced by students as a result of the teaching session. Teachers involved in creating the TLRPs were also asked to keep diaries of their activities to monitor the development process.

A centralised approach was adopted, with responsibility for the development of evaluation tools and guidelines, cataloguing, data analysis and reporting resting with one site (Edinburgh). This had the advantage of standardising the process - all of the sites at which evaluation data were collected used the same evaluation tools enabling comparisons to be made between data from several sites. It also reduced the time commitment of the staff at test sites who would administer the tools and collect the raw data, thereby increasing and encouraging their level of participation in the process.

The process was managed through the appointment of a research assistant and this proved to be a major benefit. She contributed to the development of a range of evaluation tools, took responsibility for disseminating these to TLRP development teams, provided telephone advice, collected all ‘raw’ data, set up appropriate databases, carried out all of the analysis, produced draft reports, chased up missing data. She has also carried out most of the structured interviews to date in phase 3 of the evaluation cycle.

Evaluation Cycle

The 3-year project involved the development of at least six TLRPs in each year. Evaluation of each is a three-stage process, typically, each stage taking place in successive years. Thus, full evaluation of all of the TLRPs developed over a three-year period will take six years. The evaluation cycle for each TLRP is shown in Fig. 1.

Fig. 1 The TLRP evaluation cycle, a 3-stage process employed to evaluate the TLRP and the process of development. In year one development teams, each consisting of academic teachers from 2–6 institutions, are formed to develop a draft version of a TLRP. During this first year one or more members of the development team will evaluate the TLRP with students and staff in their university. During year two a similar evaluation is undertaken by one or more pharmacologists who are not members of that TLRP development team but who are members of the project consortium. Feedback from stages one and two of the evaluation is used to modify the TLRPs to produce a final product which is then made available to other universities through the British Pharmacological Society who also distribute the pharma-CAL-ogy software. Phase three evaluations are then undertaken with staff and students from universities, world-wide, who purchased one or more TLRP’s and who agree to participate in the evaluation process. These will be ‘naïve users’, often totally unfamiliar with the project and it is they who will provide the most useful product evaluation data.

Evaluation tools and guidelines

A set of evaluation tools and guidelines was developed to standardise the data gathering process. Each site developing or evaluating a TLRP received an evaluation pack consisting of:

• Staff pre- and post-TLRP questionnaires (masters for photocopying and provided online)

• Students pre- and post-TLRP questionnaires (masters for photocopying)

• Guidelines for structured interview sessions

• Guidelines for developer log/diaries

• Pro-forma log sheet

• Diskette containing .doc (Word 6) and .rtf (Rich Text Format) files of the above.

Evaluation Tools

Three types of evaluation tools were developed: questionnaires, indicative structured interview questions, and knowledge assessments (Triangulation evaluation; Webb et al., 1996). Monthly proforma log sheets were also used to standardise the data collection for the process of TLRP development.

Questionnaires

Draft questionnaires were developed in year one of the project and piloted and validated with year one development teams. Various question types were used. Single and multiple response tick boxes (closed questions) and scaled response questions were used to collect course/personal information (e.g. which university, course, year of study, age), experience of using IT, and experience of using CAL programs in pharmacology. Five point (strongly agree to strongly disagree) Likert attitude scales were the predominant question type and aimed to assess the attitudes and perceptions of the staff and students before and after a CAL/TLRP resource had been used in a teaching session and to draw some comparisons with traditional teaching methods such as lectures and tutorials. There was a single open-ended question for further comments. These questions were grouped into 3–4 sections according to category. For comparative analysis negative and positive tense questions were used.

The pre-TLRP student questionnaire comprised 3 sections and a total of 35 questions. The post-TLRP student questionnaire comprised 36 questions and differed slightly from the pre-TLRP questionnaire.

Similar pre- and post-TLRP questionnaires were developed for teaching staff. There were minor differences e.g. they included a fourth section in the pre-TLRP questionnaire designed to ascertain which teaching tools/methods they used in their own pharmacology teaching. The post-TLRP questionnaire was similar to that for students but with the focus of the questions changed to ascertain the teacher’s perspective (a total of 33 questions). Each development site was responsible for administering and collecting the completed questionnaires and returning them by post to the central evaluation site.

Responses to the questions were mostly pre-coded, however a few open question responses were coded later. Data from paper based questionnaires were logged manually into the statistical analysis software package SPSS v10.1 which was used to handle and analyse the data. For questionnaires submitted online the information was stored in a Microsoft Access database and exported to SPSS for subsequent analysis.

Structured interviews

Structured interviews were used to explore issues in greater depth. Prior to the TLRP trial, data was gathered from students and teachers about current teaching methods and the factors they perceived might influence how CAL resources were used in their institution — baseline data. The same sample of students and staff were interviewed again to measure changes in their attitudes and perceptions towards CAL resources following the use of a TLRP. Guidelines on how to conduct the sessions together with a number of suggested questions were provided to those carrying out the evaluation. The taped interviews were analysed and transcribed using the notes taken by the note-taker present during the interview to supplement the transcriptions

Knowledge assessments

These were developed as part of each of the TLRPs and employed during/following the teaching session in which the TLRP was used.

Developers log/diary

The process of TLRP development was monitored using a monthly diary/log in which the TLRP-development teams recorded personal project-related activities. Again guidelines to standardise the process and proforma log sheets to facilitate data collection were developed.

Results and Discussion

The results of phase 1 evaluation of 12 TLRPs, (representing staff and students from 15 universities) and phase 2 (8 universities) are presented and some of the findings will be discussed. Phase 1 data was considered to be extremely important in ensuring the quality of the TLRPs developed. Phase 2 data adds some value but the major summative data will be gathered in phase 3 and is not available for all TLRPs at this time. Student users from various undergraduate and postgraduate courses (e.g. MBChB (undergraduate medicine), BSc Biological Science, MSc and PhD) have participated in the evaluation. Data from only a relatively small number of teachers has been collected.

Baseline data

Baseline data represents views of students and staff before they had experienced a teaching session using a CAL/TLRP. It was gathered by means of both pre-TLRP questionnaires (results presented below). It was planned to use structured interviews to explore issues in more depth but only a relatively small number (nine: four student groups; five staff groups) were actually conducted with staff and student groups during year one. To date year-2 and year 3 development teams have provided no additional baseline data. In general the guidelines for structured interviews have been followed although recommended group sizes (5–8) have often been smaller (usually 2–5) than those which were recommended.

The number of interviews was small and it is difficult to draw any meaningful conclusions. However, a number of comments were common to all interview groups and are summarised below:

• both staff and students

  • awareness of e-resources and experience of their use varies widely;

  • CAL programs are perceived to be valuable learning and teaching resources e.g. help to reinforce understanding particularly of dynamic processes and provide clear visuals of mechanisms and interactions;

  • e-learning has many advantages over traditional teaching methods e.g. enables self-paced, time and place independent learning.

• Students expressed concerns about:

  • potential loss of contact with tutors;

  • lack of clarity about learning objectives.

• Teachers reported:

  • concerns about the difficulty of resource discovery and suggested that measures to promote awareness of CAL resources would be useful particularly if they also had access to independent reviews or demonstration copies;

  • that staff development workshops on how to integrate CAL programs into courses would be useful.

Pre- & Post TLRP questionnaires

Questionnaire data has been collected from twelve of the twenty TLRP development teams (student: pre-TLRP n=745, post-TLRP n=620; staff: pre-TLRP n=4, post-TLRP n=6). The number of completed staff questionnaires is extremely disappointing and probably too small to enable any useful conclusions to be drawn. Currently, only 4 TLRPs have been evaluated beyond stage one.

The Likert scale questions were common to both pre- and post-TLRP questionnaires for both staff and students. The % figures quoted below represent the aggregate of those reporting strongly agree/agree or strongly disagree/disagree. Analysis of these data showed a high level of agreement between the data collected from different sites. Analysis of the pre-TLRP questionnaires suggested that the student perceptions of e-learning were largely positive and after using the CAL/TLRP resource in this study they became more positive. Students felt strongly (92%) that CAL resources should always be available, 63% reported that if they were available over the web (i.e. accessible off-campus) it would motivate them to do more independent study. Only 25% indicated that they were worried about using technology before they participated in the CAL/TLRP session and this figure fell to 5% after the teaching session. 54% were positive about the flexibility (time and place independent) of this mode of learning.

Analysis of the post-TLRP questionnaires showed that students were very positive about the quality of the TLRP (content was pitched at an appropriate level (64%), was relevant (75%), tasks were challenging (58%)), graphics and navigation features were good (60%), Most (80%) were very positive about the flexibility offered in terms of time, place and duration of use of the resources (a statistically significant increase compared to the 54% pre-TLRP

-Mann-Whitney test, P<0.005). There was also a significant pre to post difference in the responses to the notion of whether they preferred to work alone or in pairs. Before the teaching session around 50% felt they would prefer to work alone while after the teaching session, in which most did work in pairs, around 66% expressed a preference for this mode of study. Before (62%) and after (59%) the session a majority of students felt that CAL resources should supplement rather than replace traditional teaching. Before the teaching session 62% thought they would feel ‘cheated’ by being deprived of tutor contact for that session, after the session this figure fell to 46%. Perhaps surprisingly, given the findings of other workers (e.g. Hughes (2001)), almost 40% said they would use the resources even if they were not assessed.

Some responses were less positive. Over 60% felt that working independently with the CAL/TLRP was not enjoyable. Over 40% preferred traditional teaching methods and over 60% would not like to see a greater level of replacement of traditional teaching with resource-based learning. Some of the comments suggested that the CAL materials contained too much factual content (70%) though they felt that the number and variety of tasks was reasonable (57%).

Given the low return rate of staff questionnaires it is difficult to draw any meaningful conclusions. However, those who did return reported that they found the pre-prepared marking schedules particularly useful in saving them time in assessing students work; the MS Word format of the TLRPs made modifying them for their own use simple; and the group activities were useful in promoting student interaction. Some reported that technical difficulties with the network and computers had caused some problems. Though they recognised the advantages of CAL programs, they expressed the view, shared by the students, that these should be used as an adjunct to traditional teaching rather than as a replacement. They also reported that using the TLRP helped to reduce the time usually taken to prepare teaching materials for a class of this type by 60–98%. The potential for some wet-practicals to be replaced with computer simulations was also recognised.

Knowledge Assessments

Knowledge assessments have been received from 4 sites (163 students). The mean marks (%) were for TLRP 5 - “Drug Targets”;” G-Protein coupled receptors” 76 ± 2.1(n =42) and 53 ±15.7 (n =55);TLRP 16 - “Rat Blood Pressure” 61 ± 12.9 (n=60) and TLRP 6 - “Introduction To Drug Metabolism” 58 ± 7.14 (n=6). It is not possible to directly compare these data with similar data assessing the ‘value’ of traditional teaching. However, the teachers’ perceptions were that students had attained similar levels of knowledge and understanding as previous cohorts of students who had been taught using traditional methods.

Diaries/development logs

Of the 20 development teams, 2 had representation from 6 institutions, 4 from 5, 9 from 4, 2 from 2 and 3 from 1. This collaborative approach worked well with different teams developing their own internal dynamics, working patterns and responsibilities. Year 1 development teams tended to be larger and some of the year 2 and year 3 TLRPs were developed by sub-sets of the year 1 development teams. Nine of the twenty development teams completed the proforma logs. A few sent them in each month as requested but it was more usual to receive several months logs at a time when it was apparent that these had not been completed as the project progressed.

The mean TLRP development time for year 1 teams was 128.9 h (range 35.2 – 188.5h); for year 2 the mean was much lower at 41.3h (range 11.5 — 98.4h) and for year 3 the mean was 43.3h (range 23.5 — 63h). The reduced time for development of year 2 and 3 TLRPs may be explained by the fact that some year-one development teams also developed further TLRPs in year 2 and 3. Their experiences in year one clearly enabled them to develop subsequent TLRPs more rapidly. Also, staff workshops held at the end of year one enabled the year 1 development teams to pass on their experiences to year 2 development teams and this too may have contributed to the faster development times observed in years 2 and 3.

Clearly, for some staff this was a very rewarding activity which they were keen to continue. The logs revealed several types of activity: collating content material; developing and subsequently modifying drafts in response to feedback from the team; producing student exercises, assessments, and marking schedules; formatting the TLRP (access to secretarial support was a very positive factor for at least one development team); discussing the teaching plan i.e. how they would integrate the CAL session into an existing course; setting up structured interviews to collect baseline data; conducting stage one evaluation of the draft TLRP.

A number of positive motivational factors were also highlighted: signs of significant progress; collaboration with group members from other universities; being part of a team gave them more confidence in the product and a desire to see how the students/teachers found it; positive feedback from students and staff who used the TLRP; good knowledge assessment results.

Conclusion

At this stage it is difficult to make definitive statements about the usefulness of the strategy developed by this project for embedding CAL resources into courses. The academic teachers involved in this project were already enthusiastic about using CAL materials in their teaching and the fact that they then spent considerable time developing TLRPs is likely to ensure that, when they use them to support their own teaching, the teaching session is presented in a very positive manner, and will be received well by the students. All of them found the development of the TLRPs to be a very demanding, time-consuming but intellectually-satisfying experience and all of them have translated their experiences into their own teaching. However, the true test of the strategy is whether institutions not involved in the project are able to use the TLRPs ‘off-the-shelf’, to support the integration of specific CAL resources into their courses. Data relating to this stage of the evaluation process (stage 3) is in the process of being collected.

Students were very positive about the quality of the TLRP support materials and they recognised the flexibility that this mode of learning offered them. Most preferred to work in pairs though it is also apparent that some prefer to work on their own. A significant number expressed the view that this mode of learning should supplement rather replace traditional teaching, they valued contact with a tutor and over 50% found independent learning less interesting than traditional lectures. It is interesting that although there were preferences expressed about many of these issues very often it was not clear cut. Clearly different students have preferred learning styles and neither approach satisfies all. An approach which blends the best of traditional methods with the best of e-learning is likely to prove most successful.

The outputs of this project have made it easier for pharmacology teachers to use e-resources in their teaching and thus encouraged the use of e-learning. Notably, the project has:

• provided support materials which will reduce the burden on staff by:

  • saving them considerable time in preparing teaching materials;

  • reducing their student contact hours in some instances

• provided exemplar teaching and learning resources which previously did not exist in many courses. These have proved to be acceptable to both staff and students, achieve the desired learning objectives and enable independent student learning;

• increased teacher awareness of the benefits of using CAL in teaching;

• provided a means of successfully integrating existing CAL resources into courses thus making the investment in those CAL resources more cost-effective;

• increased the acceptance of CAL resources developed elsewhere.

A major barrier to the use of CAL resources is a reluctance by teachers to use materials developed outside the institution (’not invented here’). The development of a ‘wrap-around’ at least goes someway to overcoming this problem as it personalises the materials and gives ‘ownership’ of the new resource (CAL plus TLRP) to the teacher. Creative teachers can use this approach to develop good teaching and learning materials from relatively poor software.

The evaluation strategy devised for this project and its subsequent implementation met with reasonable success given the difficulties of standardising data collection over several sites. Clearly some development teams complied with the evaluation strategy to a much greater extent then others. Discussions with the development teams revealed that although they appreciated the value of evaluation it was of low priority compared to TLRP development. Some team members found it difficult to provide an impartial opinion and felt it would not be appropriate for them to take part as, through their involvement in the project, they were already enthusiastic users of CAL and probably not representative of the target user population.

The centralisation of the process was an important factor and it is unlikely that a similar quantity or quality of data would have been collected had the development sites been left to organise ‘local’ evaluations.

Data collection is expected to be completed early in 2003 and data from stage 3 evaluation will be particularly important in drawing meaningful conclusions about the success of the strategy this project has adopted. However, the evidence so far suggests that there is considerable potential to harness technology to enhance or even replace some aspects of traditional course delivery in pharmacology.

Acknowledgements

This project was funded under phase 3 of the Teaching and Learning Technology Programme (Project 83: Implementing technology-based teaching and learning in pharmacology).