The Quantitative Effect of Students Using Podcasts in a First Year Undergraduate Exercise Physiology Module

Abstract

This study reports the quantitative effect of students using podcasts in a 1^st year undergraduate exercise physiology module. From a cohort of 70 students, 50 volunteered and completed the study. Using a pre-post random allocation research design, students were allocated to either a podcast group (PG) or control group (CG) based on a 32-question multiple-choice exam. The PG then listened to six podcasts over six weeks, while the CG were provided with an exact transcript of the podcasts in printed form to ensure that both groups were provided with the same content. After six weeks, both groups were re-examined using the same test. Data were analysed using the effect size statistic and 90% confidence intervals. The CG improved their exam performance by 43%, whereas the PG improved by 46%. The difference between the groups on the post-test was a mean effect size of 0.19 (90%CI: −0.16 to 0.53 [trivial to positively small]). There is almost no chance that the true effect in the population is harmful. The results of this study suggest that using podcasts provides little quantitative benefit for students over and above written text when learning exercise physiology.

Keywords:

• podcasting
• exercise physiology

Introduction

A ‘podcast’ is simply an on-demand media file that can be automatically downloaded from the web to a computer or portable media player, such as an iPod, for listening offline. Given this ability to listen to podcasts on a mobile device, it is important that we study their effectiveness, as portable devices are emerging as a major technology within e-learning (Fischer et al, 2003). While the technology to create and publish on-demand media such as an audio MP3 file has been with us from the advent of the internet, the popularity and uses for podcasts are relatively new, following the introduction of the MP3 media format and development of large capacity portable players such as the Apple iPod range. Moreover, by making use of RSS standards (Kelly, 2005), users ‘subscribe’ to the podcast thereby ensuring that the latest content is automatically downloaded to their device when the user is next online. The popularity of podcasts has exploded over the last two years probably due to widespread availability of broadband and increasing access to the internet at home, with more than 100,000 podcasts listed on the iTunes Store as of August 2007 (Apple, 2007b). Mainstream media such as the BBC now publish many of their radio shows as on-demand podcasts. The popularity of the iPod has certainly contributed to this, with over 100 million iPods sold since 2001 (Apple, 2007a). However, it is also the creation of podcasts that has become easier to achieve. Software such as Apple’s GarageBand and other audio editors such as Audacity have made the process of recording and editing audio much easier, thereby allowing enthusiasts with access to the internet (including students and lecturers) to create and publish their own podcasts. For a review of podcasting see Campbell (2005).

Young people have grown up with computers and technology. Sometimes described as the Net generation (Oblinger and Oblinger, 2005), they are assumed to be digitally literate and accustomed to using the Web for supporting learning. Given the creative freedom this allows, podcasting is being embraced by educators, providing another method for supporting students in their learning. The possible values of podcasts as an integral part of students’ learning ‘toolkit’ are that they are mobile and easily accessible, popular and non-threatening. The technology is probably not a barrier to learning as students are familiar with using MP3 players for downloading and listening to music. For example, Newnham and Miller (2007) found that 88% of information systems students agreed or strongly agreed that accessing podcasts was easy. However the question of whether these digitally literate students will be willing to embrace the use of mobile technologies within the educational context requires further research. Campbell (2005) believes that students will be comfortable with using mobile technologies for learning, but recent studies from the University of Washington and RMIT University have shown that 81% and 85% of students respectively listened to podcasts on their personal computers indicating mobility was not the prime motivator in using podcasts in their learning (Lane, 2006, Atkinson et al, 2007). It is not the purpose of this study to assess the mobility of using podcasts but the effect this form of media has on exam performance.

While studies examining the use of podcasts have contributed valuable information on the many qualitative aspects of their use (Belanger, 2005, Lane, 2006, Bell et al, 2007), recent advice has suggested that more studies should report changes in desired (quantitative) learning outcomes, rather than just qualitative measures (Michael, 2006). To that end, we are not aware of any quantitative studies examining the effect of podcasts on student achievement. As universities wish to promote independent modes of study and increase learner autonomy the aim of the current study was therefore to examine whether students could enhance their knowledge of exercise physiology, leading to a quantitative improvement in exam scores by giving them greater control over aspects of their own learning in terms of time and place to learn.

Methods

Subjects

Students from a 1^st year undergraduate exercise physiology module volunteered for the study reported here. Students were initially provided with a verbal explanation as to the nature of the study, including risks, benefits and the time commitment required. From a total of 70 students in the cohort, 60 students volunteered to take part. Due to some students failing to take the post-test the final sample size was 50, with 23 in the podcast group (PG) and 27 in the control group (CG). The students were a mix of sex (33 males, 17 females) and age (range 18–22 years).

Research Design

The study used a pre-post random allocation research design where students were randomly allocated to either the PG or CG. Those students involved in the study were initially tested using a formative multiple-choice exam (explained in more detail later). Students in the PG (n = 23) were then asked to listen to a series of six podcasts over six weeks that supplemented their regular teaching and lecture material. Students in the CG (n=27) were provided with an exact transcript of the podcasts in printed form to ensure that both groups were provided with the same content. Consequently, the difference between the groups was the method of content delivery, rather than the content per se. After six weeks, all students then repeated the assessment using the same questions and exam format as used for the pre-test. The study was approved by the University of Cumbria Ethics Committee.

Podcasts

A total of six podcasts produced for the study were recorded in a ‘radio’ style format incorporating a question and answer discussion between the two authors, together with jingles, music, section breaks and images. Podcasts ranged in duration from five to 14 minutes. Each podcast related to the study of the muscles energy systems being taught during the six-week period. The six podcasts were titled (1) Energy Systems, (2) ATP/PC Energy System, (3) Lactic Acid Energy System, (4) Aerobic Energy System, (5) Recovery, and (6) Study and Exam Tips. Podcasts were recorded and produced using GarageBand 3, part of the iLife suite of applications (Apple Inc., Cupertino, USA). The podcasts were ‘enhanced’ with the addition of static images that could be viewed within software such as iTunes and also on iPods. The static images were designed to create an ‘imagery hook’ upon which the student could position the associated discussion. Chapter markers were placed at appropriate places in each podcast to allow for easy navigation within iTunes or an iPod.

Podcasts were available to the students in the PG to subscribe or download from a dedicated website and also on the institution’s virtual learning environment (Blackboard™). These sites were available only to students in the PG. Students in the PG were instructed to download and listen to one podcast per week. They were able to listen to each podcast as many times as they wished on their computer, iPod or generic MP3 player. The podcasts were made available in both AAC and MP3 file formats for viewing and listening in iTunes on a computer and/or an iPod/MP3 player. Due to technical limitations we could not monitor the number of times each podcast was listened to, or which version of the podcast was listened to. However, we did obtain a verbal acknowledgment from each student in the PG that they had listened to all six podcasts prior to taking the post-test. Additionally, we had confirmation from those in the CG that they had read all six transcripts. Following the post-test all podcasts were made available to all students enrolled on the module.

Teaching

The PG and CG were provided with the same instruction over the six-week period as was practically achievable. The 2^nd author (TB) was the module leader and conducted all lectures during the period of the study. These lectures were one hour in duration, presented to all students enrolled on the module and consisted of traditional slide-based presentations in a lecture theatre. However, two tutors (TB + 1) were used to teach the laboratory sessions. The laboratory sessions were conducted in the School’s Exercise Physiology Laboratory with a one-hour duration. They consisted of both experimental and tutorial-based sessions. Consequently, all students (PG and CG) received 12 hours of face-to-face instruction over the six-week period. Within the School of Sport, students are encouraged to engage in approximately four hours of independent study per week for each module. Students enrolled in the module were required to have a minimum of C at GCSE science, but an A-Level in a science discipline was not a requirement.

Exam

Students’ performance was evaluated by a 45-minute formative exam. The exam consisted of 32 multiple-choice questions that examined the students’ knowledge of exercise physiology concepts such as the energy systems, fatigue and recovery. Examples of the questions are listed in Table 1. The six-week period of study was during semester 2 of the 2006/7 academic year. The 1^st year undergraduate exercise physiology module is a full-year module worth 20 credits. A pass mark of 40% was used, which is normal in most UK higher education institutions.

Table 1 Samples from the multiple choice question exam administered both before and after the podcasting intervention

Question	Possible Answers	Topic Assessed
The prime function of the energy systems is to?	Convert nuclear energy to mechanical energy Convert mechanical energy to kinetic energy Convert chemical energy to mechanical energy Convert mechanical energy to chemical energy Convert kinetic energy to mechanical energy	General energy metabolism
Creatine supplementation may improve performance in repeated sprint activities as it:	May improve the recovery of PCr in the muscle cell Reduces creatine in the kidney Improves fast-twitch muscle fibres Reduces training intensity Stimulates glycolysis	Phosphocreatine Energy System
Lactic acid rises in muscle and blood during intense exercise due to:	Failure of the cardiovascular system A mismatch between production and removal of lactic acid A lack of oxygen A failure to drink fluids before training Not enough carbohydrate in the muscle	Lactic Acid Energy System
The electron transport chain contains a series of ______ and ______ reactions.	Oxidation and reduction Glycolytic and aerobic Anaerobic and aerobic Hydrolysis and hydrophobic Simple and complex	Aerobic Energy System
The recovery oxygen uptake reflects which of the following effects of exercise?	Anaerobic metabolism Respiratory adjustments Thermal adjustments Circulatory adjustments All of the above	Recovery

Statistical Analyses

Outcomes for the dependent variable of ‘exam score’ are first presented as means and standard deviations for the PG and CG. Differences between means are presented first as the original exam score, then as an Effect Size (ES). We show the precision of these measures as 90% confidence intervals, which define the likely range of the true value in the population from which we drew our sample (Curran-Everett et al, 1998, Curran-Everett and Benos, 2004, Batterham and Hopkins, 2006). We agree with these authors that the estimation of the ES, and the precision of that estimation using confidence intervals, is more important than hypothesis testing using p values. Moreover, a confidence interval that does not include zero is analogous to stating statistical significance (Curran-Everett et al, 1998, Curran-Everett and Benos, 2007). The smallest practically important change (either positive or negative) in exam score was assumed to be an ES of 0.2, with the full scale of effect size descriptors being trivial (0–0.2), small (0.2–0.6), moderate (0.6–1.2), large (1.2–2.0) and very large (2.0–4.0) (Hopkins, 2002). An effect was deemed ‘unclear’ if its confidence interval crossed both positive and negative ‘trivial’ thresholds. For example, if the confidence interval for an effect ranged from negatively small to positively small, then the effect would be labelled as unclear, as it is not clear if the effect is positive or negative. We also present the chances (as a percentage) that the ES is trivial, small, moderate, large or very large. All calculations were made using a dedicated spreadsheet designed by Hopkins (2003).

Results

The mean±SD for both the CG and PG are shown in Figure 1. There was a small difference between groups on the pre-test (CG − 12.5 ±4.4 v PG − 13.5 ±3.7; ES = 0.23). The post-test showed that the CG and PG improved their exam scores to 17.9 ±4.9 (43% improvement) and 19.7 ±3.4 (46% improvement), respectively. This meant that the PG improved their post-test exam score by 0.8 marks more than the CG (90%CI: −0.7–2.3). This difference as an ES was 0.19 (trivial), with a 90% confidence interval of −0.16 to 0.53 (trivial to positively small). That is, the true (but unknown) effect of listening to the podcasts when extrapolated to the population ranges from trivial to positively small. Moreover, we have calculated that the chances of the effect being trivial is 49% (possible), or being positively small is 48% (possible). The chance of the effect being negative (harmful) is 3% (very unlikely).

Figure 1 The MCQ exam score for both the control group and podcast group before and after the six-week podcast intervention. The difference between the groups on the post-test was an ES of 0.19 (trivial [90%CI: −0.16 − 0.53; trivial to positively small])

Discussion

Both the PG and CG improved their exam grades by 46% and 43% respectively. This study indicates that supplementary material delivered either in the form of a podcast or written transcript enabled the students to improve their knowledge of exercise physiology. The main finding of a mean ES of 0.19 (trivial effect) suggests that the use of podcasts in an undergraduate setting may not result in a worthwhile improvement in student achievement over-and-above the use of written material. It must be remembered that the calculated ES is the net improvement in the PG, and therefore represents the size of effect above the ‘normal’ amount of learning (the CG improved by 43%; the PG improved by 46%). However, based on the confidence interval there is a 48% chance that the effect is worthwhile, although that effect would be ‘small’ at best. There is very little chance of a harmful effect from students using podcasts. Given this, it could be argued that students have nothing to lose by using podcasts for learning, as there is virtually no chance of a harmful effect and some chance (48%) of a ‘small’ advantage. However, this must be weighed against the time and resource commitment required to develop and support the use of podcasts for students. As students spend most of their time with printed texts it may not be surprising that there was little net effect on exam performance in the PG.

In the current study we have used podcasting to support formal learning that is otherwise delivered on paper or through web resources. In using podcasts we have aimed to make the process of learning more active and engaging by including multimedia. Greer (2006) describes the need to make the content relevant to the context of the individual learner and stimulate learner interest. In designing these activities we have attempted to adapt the content to the medium in a way to motivate students and provide a flexible ‘just in time, just enough’ format (Pritchard, 2004). Although our aims were admirable we did not involve the students in the design of the content in the podcasts. Brittain et al (2006) state that actively involving the students in the evaluation and design of a learning technology project enhanced the success of podcasting their lectures. Similarly Kirkwood and Price (2005) report that students need to know not only what they are supposed to do, but why they are expected to do it and how it will enhance their learning if they are to engage with new mobile technologies.

Furthermore, many researchers have stated that learning is unlikely to be improved by the mere application of a new technology (Alexander and McKenzie, 1998, Kirkwood and Price, 2005). It is important to evaluate the context within which the learning took place. The tests used in this study were formative in nature and hence students’ perceptions of the importance of them may have been reduced as they did not contribute to their final mark. Research by Bolger and Somech (2002) claims that students are motivated to complete their degree with the minimal effort and Kirkwood (2003) states that assessment that counts towards their final grade influences the choices they make when engaging in learning activities. It may be that although all students stated that they listened to all podcasts on at least one occasion, the current intervention may not have been enough to make a larger impact on learning than reading the same material.

The nature of the podcasts allowed students to repeatedly listen to the discussion and watch images associated with those concepts. One of the most-used and successful methods of improving memory is through the use of imagery association. That is, associating an image with a piece of data or information appears to be an effective method for learning (Reed and Hoffman, 1986). However, our research allowed students to listen to the podcasts in both MP3 or enhanced AAC formats. Clearly there is a need in future studies to provide a clear classification framework for the variety of podcasts that are available to use with students. The style of podcast, its duration, content and format presentation may all have different effects on learning and thus exam performance.

The results of our study appear to be at odds with other studies examining the use of podcasts and/or iPods (Belanger, 2005, Brittain et al, 2006, Lane, 2006, Rudel, 2006, Bell et al, 2007, Dale, 2007), although none of these studies directly compared the use of podcasts with the same content delivered in another form (e.g. written transcript). For example, at Duke University a number of benefits for using iPods have been identified, including greater student engagement and interest in teaching activities (Belanger, 2005). While Belanger (2005) did not study the use of podcasts per se, the two are inherently linked. Moreover, the Duke University experience might suggest that technologies such as iPods and podcasts may offer more subtle benefits to students than that which can be measured in an exam.

For future research it may be prudent to involve students in designing and producing podcasts to motivate them to engage with the technology. This combined with content linked to a summative assessment, which contributes to a final grade may further encourage students to realise how the use of technology enhances their learning. If students see that it has a benefit in their final exam this extrinsic motivation may encourage greater engagement with new mobile devices. An accurate weekly recording mechanism to monitor use of the technologies such as a diary may also prove beneficial in future research.

With the predicted growth in the use of podcasts for recreational purposes more research is needed on whether students wish to adopt the use of mobile technologies within their education. At present a majority are using personal computers to listen to educational podcasts indicating that mobility is not the prime drive for using this media format (Brittain et al, 2006, Lane, 2006, Bell et al, 2007). A clear framework for the variety of formats that podcasts are delivered in is required and how these formats influence pedagogy. The present study reinforces the idea that learning in higher education will not be improved simply by the application of a new technology. It is crucial that new innovations take into account pedagogical design, the learning context, prior student experience and institutional support mechanisms.

Conclusion

The current study has demonstrated that undergraduate exercise physiology students have gained a direct benefit from being provided with supplementary material in the form of printed text and podcasts. However, the use of podcasts provided little additional benefit over and above the printed text. That being said, listening to podcasts appears to have no detrimental effect on learning and there is a chance (48%) of a worthwhile effect. Consequently, individual academics or departments must weigh this up against the time and resource commitment required to develop and support the use of podcasts for students. While the use of podcasts appears to provide little advantage over written text, there may be other ‘qualitative’ benefits that arise from the use of podcasts, and these have been shown in other studies. Given the infancy of podcasting and its use within education there are a range of questions that future studies need to address. However, based on the current study future research needs to examine the involvement of students themselves in the creation of podcasts and also the effect of linking this creation with summative assessment.

Acknowledgements

This project was funded by the Centre for Bioscience. We would like to acknowledge the contribution of our students to this project.