A critical examination of the time and workload involved in the design and delivery of an e-module in postgraduate clinical education

Abstract

Background: Although there is increasing pressure on Universities to implement e-learning, this ‘glorious revolution’ has been met with disappointing results and universities have struggled to engage academic staff, who are major stakeholders, with its use. Although literature suggests online teaching adds to traditional faculty workload, information surrounding the actual ‘cost’ to individuals is sparse. For academics involved in postgraduate clinical education, it is even more incomplete. Involvement can be a risky undertaking for academics unfamiliar with the resources required.

Aims: This study outlines staff resources required to create an e-module for busy, practicing clinicians.

Method: Data (web analytics, email traffic, and work logs) was collected and statistical analysis performed outlining time involved, work patterns and responsibilities.

Results: Data analysis revealed 75% of academic time occurred out of normal office hours. Sixteen total staff hours (12 planning and four delivery) were required to support one hour student online activity. Technical responsibilities were essential throughout, but unpredictable.

Conclusions: Universities struggle to engage staff with e-learning due to its unrecognized and (many academics believe) unsustainable workload. Avoiding ‘traditional’ workload assumptions that are inaccurate, this study provides academics and managers involved in clinical education clear guidance and an increased understanding of workload with a goal to inform practice.

Introduction

Although, one of the claims of e-learning is that it will save time and money (Beetham & Sharpe 2010), e-learning is consistently reported to be more time consuming and expensive than traditional teaching (Laurillard 2007; Mancuso 2009; Major 2010). Additionally, there appears to be a lack of clear guidance (Ryan et al. 2004) and organizational support (Curtis 2001) for faculty when developing and implementing e-learning. To further obscure the picture for those involved in clinical education, the majority of research on e-learning has focused on undergraduates or children and has not focused on health professionals as learners (Cook 2009). The actual cost of e-learning is not well understood and literature offers little help for academics and has almost no impact on practice (Laurillard 2007). How much does embracing technology in education cost academics? Rumble (2001), when discussing the cost of e-learning, suggested that placing monetary values on teaching activities and resources being used was unreliable as these numbers have to be related to complex financial structures of different institutions. A more transferable or practical solution that could impact on practice may be to look at time involved. If academics are expected to create effective e-learning materials for clinicians, how can they plan workloads, and be empowered to argue for resources if they are not aware of or prepared for the time commitment required? Research that informs and shares good practice is one way. However, according to Medical Teacher, the most quoted article from 2009 was David Cook's ‘The failure of e-learning research to inform educational practice, and what we can do about it’. In Cook's (2009) article, he suggested we should be focusing on creative quantitative research in e-learning, for example, the exploration of monetary and time costs of technologies.

Table

Practice points

Sixteen hours of staff time (11 academic, 1.5 admin, and 3.5 technical) were required to develop each hour of online student activity. Seventy-five percent of academic contributions occurred outside work hours. Almost four hours of academic moderating and facilitating time were used per hour of online student time. Academic and administrative roles were consistently essential throughout the development and delivery of the module. However, technical demands varied which suggests the need for a flexible work pattern. We are changing the way we think students are learning, therefore, we must change the way we think we are teaching.

This study is a direct response to these arguments. The aim was to outline the time involved, responsibilities, patterns and shifting workload of a University team involved in a module written for full time practicing clinicians in order to inform frontline academics and as a result, institutional practice.

Background

In 2011, Newcastle University's Master of Clinical Education program (http://www.ncl.ac.uk/medev/postgraduate/clined/index.htm) delivered its first fully online module entitled ‘Utilizing Technology in Clinical Education’. Feedback from students was overwhelmingly positive and there was both a 100% pass and completion rate. However, the experiences of the academics (of which I, the author was one) followed an arduous path, both in development and implementation. From the outset, the main difficulties encountered by the entire team (academic, technical, and administrative) included time involved, responsibilities and distribution of labor. These are common themes in the development of distance learning, however, literature addressing these is surprisingly sparse (Rumble 2001; Laurillard 2008). The school had no formal e-learning strategy or plan with which to work. The academic did not have the knowledge, experience or confidence to make firm demands on workload planning, work release, out of hours expectations, or administrative and technical support. Similarly, informal feedback from the other members of the team (administrative and technical) suggested that the time they required and protecting that time was a major challenge for which it was difficult to plan. None of the team was prepared. This study investigated the process more thoroughly using quantifiable data, in terms of email traffic, work records and web analytics. By sharing this model of time and work patterns, it may inform academics, technicians, administrators, and managers embarking on a similar initiative. Laurillard (2007) argues that innovators in e-learning should share their planning models which may enable a community of practice consisting of innovators and managers to develop and improve tools and lead to a better general understanding of ‘cost’. This was not a comparative study, although literature comparing cost of distance to face-to-face learning (Garrett & MacLean 2004) exists and varies significantly from a fourfold increase (Carlock et al. 2001) to a twofold increase (Jewett 2000) in time compared to traditional teaching. Cook (2009) argues that comparing traditional learning (lecture) to e-learning (internet) does not make sense as effects will forever be elusive because of the heterogeneity of the interventions. Laurillard (2007) argues that there is no consistency in the parameters used comparing costs of new technology with traditional methods in an institution. Researchers should resist the tendency to seek global effects of e-learning in comparison to other methods; the focus should be on studies that will inform practice (Cook 2009).

Context

When reviewing literature on e-learning, the lack of information regarding context makes it difficult to apply the results to other settings and as a result difficult to inform practice. In three systematic reviews concerning e-learning in health care professionals (Wutoh et al. 2004; Khan & Coomarasamy 2006; Lam-Antoniades et al. 2009), context was not specifically addressed and no effort was made to obtain additional information from authors. Cook et al. (2008), in a meta-analysis of e-learning and health care professionals, suggested that most of the literature surrounding e-learning failed to describe key elements of context. For this paper to be useful to others and inform practice, the context must be described in detail.

This 20 credit module contributed to a postgraduate Diploma in Clinical Education at Newcastle University, a traditional ‘red brick’ University in the Northeast of England. e-Learning often starts as small scale independent projects (Robinson 2001) as was the case with this module. At Newcastle University, e-learning has not been initiated on an institutional basis; this module was developed by an academic (me), an administrator and a technician with no e-learning experience. There were eight students enrolled on the module: seven doctors and one dentist. All of these students were full time practicing clinicians of varying grades and specialties. The module was completely off campus and asynchronous, ran from January to June and consisted of three ‘strands’. Each strand was ‘active’ and online for two weeks and consisted of independent activities, discussion forums, wikis, required reading, individual and group tasks. The activities in the strands were created, pretested and tailored toward approximately seven hours of student work time. (After each strand the students were asked anonymously how long the strand took; the average completion time was 10 hours). During the strand active time, the wikis and discussion forums were facilitated by the academic team. There was approximately three weeks break between each strand. The final assignment consisted of a 2000 word critical analysis, using personal experience and literature, of any technology-enhanced learning initiative that was relevant to the learners.

Methods

Time spent by staff, email traffic, and workload patterns were quantified. The data was approached from a positivist viewpoint and was treated as an object that could be captured and measured (Crotty 2005). The data was collected and analyzed objectively in a search for answers. Prior to the module going live, ethical approval was obtained from Newcastle University to collect workload data from the team directly and through web analytics.

Data sources

Three data sources were investigated: self-reported employee work logs, email traffic, and web analytics. These are all valuable sources of information when investigating actual practice in e-learning as the link between new IT projects at the University level and faculty is of primary importance (Christenberry & Sturgeon 1996). Work logs were the first sources investigated. According to Christenberry and Sturgeon (1996) in IT development in Universities, work procedures are misunderstood by the institution and the link between what is actually being done could be manifested through work logs and transaction records kept by employees. The second data source was a full record of staff email traffic concerning the module. Counting emails does not necessarily indicate workload; however, email exchanges form a tangible part of the transactions between staff and are easily accessed (Baumfield et al. 2008). The third data source, web analytics, provided the opportunity to explore staff work and behavior patterns in e-learning. Web analytics, defined as ‘the measurement, collection, analysis, and reporting of Internet data for the purpose of understanding and optimizing Web usage’ (Web Analytics Association 2010), have been used primarily in business to track consumer groups related to marketing efforts, but can be used as a powerful way of extracting actionable knowledge in distance education (Rogers et al. 2010). This actionable knowledge was what I hoped to gain in order to inform practice. Web analytics were used to create a concise picture of academic time requirements during the active stages of the module.

Data collection

There were three main team members involved in development and delivery of this e-module: an academic (myself), a technician and an administrator. Detailed logs were kept of time spent in any activities surrounding the development and delivery of the module. Each staff member kept a record of time spent working on the module beginning six months before actual implementation with students until the module was completed. Time was classified into one of two categories: development (all activity before the module went live to students), or delivery (all activity after the module was made live to students). These records were collected weekly and put into a shared data base and ‘themed’ as academic, technical or administrative responsibilities. All email traffic concerning the module was saved in a separate folder in Microsoft Outlook and categorized in a similar method: development or delivery. After the module was completed, these tangible e-mail records of activity were categorized by ‘theme’: either academic, administrative, or technical. To increase reliability, an independent educationalist reviewed and categorized a random sample of 30 emails into the three ‘themes’ of: academic, administrative or technical issues. There was a 100% agreement. This suggested reliability with the categorization, and also suggested the tasks and issues fell clearly into one of the three themes. With these work logs and email records, time involved, trends, and workload patterns were identified. Analytics are collected routinely within our learning support environment in web logs. Using a system administrator, and adhering to ethical standards, relevant data was extracted from the web logs to create online working profiles. Academic working patterns and access behaviors were outlined. All data collected was anonymous. Data of individual staff was not considered as valuable as that of the entire academic team, so although in some instances (e.g., when staff signed their name in the discussion forum or learning space) we could identify individuals, this was of no benefit and therefore not recorded against a name. The data was collected and analyzed and trends were identified as an academic group.

Results

The results are presented mainly visually and in chronological order. Analysis regarding development (before the module went ‘live’) are presented first. The results of analysis post-implementation or delivery (after the module was made ‘live’) to students follow. Overall comparisons are given at the end.

Development phase

There were over 600 emails exchanged between the academic, the administrative, and the technical team during the development of this module. Figure 1 shows the focus and nature of staff emails in the development phase. Administrative issues comprise the majority of the concerns. Although the team was congenial, the nature of the emails was never of a personal nature; communications were all task related. As evident in Figure 1, the technical, academic, and administrative emails appeared to follow a similar pattern. This would suggest that all three of these roles required high levels of activity throughout. The majority of these occurred within the last month before implementation. In hindsight, this was not surprising; however the team was not prepared for the actual volume of work.

Figure 1. Timing and nature of staff emails pre-implementation.

Figure 2 demonstrates the number of hours the academic, administrator, and technician took to develop each strand; as the module progressed, overall the total time commitment decreased. Figure 3 illustrates, as the module progressed, the ratio of academic time increased and that of the administrator and technician decreased. This demonstrates a change in both actual workload and patterns of work. The academic began performing more of the administrative and technical roles.

Figure 2. Staff developmental time in hours by strand.

Figure 3. Ratio of staff development time by strand.

Delivery phase

The following data was collected after the module was released and went ‘live’ to students.

Over 75% of the recorded academic time (on wikis and discussion forums) was spent outside of ‘normal’ work hours. This information concerning working patterns for staff is essential for workload planning. Figure 4 demonstrates the large volume (over 400) and focus of inter-staff emails, during delivery. Even after the module was running, the support of the rest of the team was essential; it was not solely academic work. Administrative issues appeared to be predominant, especially in the opening and closing fortnight of the module. Technical issues appeared to be less significant but occur in bursts. Overall, there was a decrease per strand in email traffic. During development, the module was pre-tested and piloted from a number of different locations (University library, public library, different private residences). Activities and software were altered and in some cases, abandoned. Although time-intensive, the team was aware that technological problems are cited by students as one of the major barriers to successful virtual learning (O’Neill et al. 2004). The functionality of the infrastructure needed to be ensured before the module was implemented – which, perhaps was demonstrated by the lack of technical issues.

Figure 4. Focus and pattern of staff emails during module delivery.

The initial section of the results dealt with ‘development’ work (before the module went live to students). The second part dealt with tasks and time invested in the actual delivery of the module. This final section of the results deals with an amalgamation of work performed in both development and delivery of the module. It is a record of the module work in its entirety.

There was a significant input of time from the entire team in the development stage. Over 200 academic hours, almost 100 technical hours and 42 administrative hours were dedicated solely to the development (not delivery) of this module (Figure 5). Each of the three strands took students approximately 10 hours to complete. Therefore, approximately 12 hours (7.3 academic hours, 3.3 technical hours, and 1.4 administrative) were used to plan and develop one hour of online student activity. The actual delivery of the module took approximately four hours staff time (3.6 academic, 0.3 technical, and 0.1 administrative) per one of actual online student activity. Combining this data suggests that for each hour of online student activity 16 hours staff time was invested in either the planning or implementation of that activity (10.9 academic hours, 3.6 technical hours, and 1.5 administrative hours).

Figure 5. Individual roles and total hours required.

Discussion

The results of this study suggest that the planning of e-learning material is time consuming from a technical, administrative and academic perspective in terms of actual workload and nature of work. These roles were essential throughout the development, were relatively consistent and immediately before delivery increased dramatically. During delivery, the technical demands decreased significantly. By careful pre-testing and piloting of the technological infrastructure, technical problems for students were avoided and the technical team had a limited student-generated workload after the module went live. Interestingly, there were very few student-generated emails concerning academic concerns before the module went online. The concerns were all technical or administrative. Even in delivery, there was a significant workload within the team, especially in the first two weeks the module went ‘live’ to students. There were over 400 inter-staff emails sent; the majority of the issues appeared to be administrative, followed by technical then academic concerns.

Not surprisingly, for an interactive, discussion based module for full-time working clinicians that was successful, this was mirrored in the academic contributions. Over 75% of all academic contributions were made on weekends or in the evening. These findings suggest not only the time involved but the pattern of work has changed for academics. Laurillard (2007), when discussing the expense of technology in education, explains ‘the more significant cost driver … is that teachers … spend their time differently’ (p. 29). Given that students consistently cite lack of immediacy and delay in tutor response as barriers to satisfaction in e-learning (Hara & Kling 2001; Petrides 2002; Vonderwell 2003), this data suggests that academics planning on addressing this student feedback, may have to change work times, habits, and traditional models of work. There is unequivocal support of distance teaching taking more time than traditional teaching (National Education Association (NEA) 2000; Brogden & Couros 2002; Howell et al. 2003; Garrett & MacLean 2004; Pirani 2004; Johnson et al. 2005; Ryan et al. 2005; Laurillard 2007; Mancuso 2009), yet the academics in this study were not prepared for the amount of work. Rumble (2001) in an inaugural lecture warned that academics, when pressured to create e-learning resources may feel a ‘hint of exploitation’ and cautioned that academics working on the course development side were particularly vulnerable to the erosion of contractual benefits. Brogden (2002) and Howell et al. (2003) agreed and suggested that one of the greatest reasons for faculty resistance in distance education was the labor-intensive and time consuming demands required to develop online resources.

This was the first time the module ran, so arguably, one would expect a higher workload than in concurrent years (Schroeder et al. 2010). Regardless, staff were doing the majority of this work out of normal work/school hours. In both formal and informal follow up discussion, this was due to: an attempt to provide timely facilitation to student contributions (69% of which occurred out of normal work hours), unpreparedness for the amount of online work and not ‘protecting’ time in the work day to facilitate the wikis and discussion. Data has recently been collected following the second year of this module being offered and the analysis will provide longitudinal insight into staff working patterns and changes.

Once the module went live, administrative issues for students appeared to be the major focus of the staff demands, followed by academic. These were relatively consistent throughout the entire delivery of the module. The technical issues, however, were far more inconsistent and issues appeared in bursts. For planning, this suggests that technical staff must be provided with a workload plan that allows flexibility in responding to tasks from both staff and students. The development time appeared to decrease as the module progressed for all staff. Arguably, as faculty becomes more experienced with e-learning, both development and delivery times may continually decrease. Regardless, this needs to be explicit and premeditated.

Alexander (2001) suggests that institutions need an e-learning plan. McPherson and Nunes (2008) agree that a plan is needed. However, they argue that e-learning requires the adoption of new models and believe that academics may not be prepared to be successful in e-environments. Reviewing critical issues in e-learning, she identified academics making ‘vociferous’ claims regarding the need for top-down strategies, appropriate levels of support, interdisciplinary collaboration, and trust. Laurillard (2008) supports this suggesting that the delivery of small scale e-learning needs a plan, collaboration, along with managerial and organizational support. Furthermore, studies of costing technology demonstrate the high costs to education and argue strongly for better planning by university managers (Schmidtlein & Taylor 2000). The author's experience, supported by the literature, is one of the main barriers and obstacles to institutional development of e-learning is an underestimation of the full cost to lecturers (Jewett 2000; Carlock et al. 2001; Brogden & Couros 2002; Howell et al. 2003; Laurillard 2007). The unrecorded hours that go into distance teaching have significant implications for workload planning models. It is essential the nature and amount or work involved is made clear at an institutional level to managers and a local level to academics. The following recommendations are a combination of the author's experience and the above data. Ideally this will inform practice and empower academics to collaborate with managers.

This study was based on self report of work from both staff and students which has inherent problems. Work was unequivocally done that was not captured as data by all staff and therefore not accounted for in the results. However, for reliability, an outside educationalist reviewed the coding of emails and there was no secondary gain for any of the staff in recording work hours. Student data was collected from a small cohort (n = 8) of postgraduates at one institution. There will be substantial variation depending on number and level of students and institutional expectations. Therefore, one must interpret the generalizability or transferability with caution. However, this is a clear record, in this setting and context of hours used and patterns of work.

Conclusions

This study will assist academics by encouraging them to ensure appropriate levels of staffing and support are available to them. Team members will be aware of time requirements and responsibilities that were barriers to the process described in this research. The results may improve communication between ‘top-down’ e-learning strategies and ‘bottom-up’ innovation and creativity. Clear information that will empower academics to outline their needs and promote confidence to break down barriers or create e-learning strategies in their institution is necessary (McPherson & Nunes 2008). This information and costing model may inform individual practice and raise awareness of the expenditure and risk to academics. This raised awareness might be a much needed instrument for change and a persuasive tool for those unaware of e-learning requirements.

Yes, e-learning is an expensive investment for academics. However, the question of whether or not to invest in e-learning is itself ‘academic’. The investment has been made. What is now required is a clear plan that addresses the needs of both academic staff and management. Guidance around workload patterns and resources requires a clear model of time involved that will ultimately inform practice on both levels. As practitioners, only through sharing informed practice can we be empowered to plan change, collaborate and avoid distance learning workload models recognized as inaccurate and unsustainable (Schofield et al. 2003).

Recommendations

1. Invest heavily in up-front technical, academic and administrative support, especially in the last two weeks before the e-module goes live. Pre-test and pilot all activities and sites and plan timings. This prophylactic care may assist in keeping technical issue to a minimum for students and help them plan workload.

2. Invest in up-front training for academics. As the module progressed the academic may take on more technical responsibilities (the ratio of responsibilities increased to demonstrate this) although, the overall time spent by both academic and technical staff decreased. If the academic had more technical expertise beginning this module, more technical responsibility could have been absorbed earlier on. More technical training should be available to academics initiating e-learning.

3. The technical demands appeared to be relatively inconsistent after the module went live and work occurred in bursts. Allowing a flexible model of planning and allocation of technical tasks is necessary. The academic and administrative issues were relatively consistent and thus, easier to plan. Ensure administrative and academic support is allocated after the module goes live and allow a plan for flexibility with technical support.

4. The percentage of online time (75%) that the academic contributed outside of working hours is concerning. One of the most imperative determinants to successful e-learning is speed and frequency of tutor contributions (Hara & Kling 2001; Petrides 2002; Vonderwell 2003). We are changing our model of learning, and therefore should be changing our model of working. Any academic involved in facilitating e-learning should change from a traditional model of working to one with work release, flexibility options, additional staffing, or paid overtime. Managers should be made aware of the timings involved and personal evening and weekend commitments that may be required to facilitate excellent e-learning experiences for students.

Acknowledgments

The author would like to thank Christopher Hodgson, a Maths student at Newcastle University, for performing statistical analysis on the data and Professor John Spencer for reviewing the manuscript drafts.

Declaration of interest: The author report no conflicts of interest. The author alone is responsible for the content and writing of this article.