A controlled comparison of teaching methods in first-year university physics

Abstract

This study presents a rigorous comparison between two teaching methods currently used in the first year of university-level physics programmes. Two groups of year 12 and 13 school students, aged between 16 and 18 years old, were given a short lecture course on elementary Newtonian Mechanics, one group attending more traditional passive-student lectures and the other attending an interactive active-student programme. The two groups were constructed to be as similar as possible in terms of relevant factors and were pre-tested with the Force Concept Inventory (FCI) and post-tested with the FCI and the Mechanics Baseline Test (MBT). Several measures of learning based on pre-test/post-test score change were employed to assess the relative effectiveness of the two lecture styles. The principal result of this study was the early onset of greater learning gains among students in the active-student programme compared to students in the passive-student programme. The particular choice of learning gain metric affected the interpretation of our results. We resolved this issue using a generalization of the normalized learning gain. Using this measure we also found improvement in problem-solving skill among females in the active-student group, but not in the passive-student group. On the other hand, males showed improvement in problem-solving skill in the passive-student group but not the active-student group.

Keywords:

• conceptual change
• force concept inventory
• gender related
• mechanics baseline test
• physics education

Introduction

As many authors have noted, students entering first year physics courses at university are often ill-prepared for study at this level (e.g. Schoenfeld 1988; McDermott 1991; Van Heuvelen 1991). While these students have generally studied physics at high school as a prerequisite for entry into university-level physics programmes, their grades are not necessarily indicative of their true conceptual understanding of the subject matter (McKinnon & Renner 1971). Many authors have observed that while many students are capable of passing traditional physics tests which assesses their ability to employ standard techniques to solve standard problems, they do not display a strong qualitative understanding of the material assessed in these tests (e.g. McKinnon & Renner 1971; Halloun & Hestenes 1985; McDermott 1991; Mazur 1997).

It has been found that students entering university-level physics courses still employ common misconceptions to explain their experiences of the physical world (Clement 1982; Van Heuvelen 1991; Mazur 1997). Clement (1982) investigated a common misconception concerning the relationship between force, velocity and acceleration, namely the belief that an object requires a force to maintain it in constant motion. Pre- and post-tests of students in a first-year university physics course were carried out to determine how many of them began the course with this misconception and how many finished with it still intact. Clement found that very little change had taken place when analysing the post-test results. While the figures vary slightly between the different test problems, it was found that only 12% to 16% of the students had amended their original misconception after attending the course. Clement is not the only researcher to experience such an effect (or lack thereof). Halloun and Hestenes (1985) obtained similar results when comparing pre- and post-test scores of students in four different university physics courses.

A number of studies have attempted to locate the root causes of these findings. Several studies have found that while students are able to adequately reproduce descriptions of concepts provided by teachers or textbooks, this does not then lead to the ability to use these concepts in novel situations. It appears that the student abilities in these cases mirror the expectations of the teaching methods traditionally employed in school and university physics classes. (McKinnon & Renner 1971; Schoenfeld 1988: McDermott 1991). McDermott (1991) also points out that information is usually disseminated to students in a top-down rather than bottom-up manner and observes that while this style of teaching may well suit the teacher, it does not necessarily suit the students.

Furthermore, the learning experiences of teachers from their time as physics students affects the manner in which they teach, i.e. teachers tend to adopt the teaching style they themselves found most helpful as a student (McDermott 1991; Thompson & Holt 1996; Wulff & Wulff 2004). Given that physics teachers are highly likely to have found learning physics fairly easy, it is unlikely that they will then teach in a style which suits students who find learning physics difficult (McDermott 1991).

Considerations such as these have led to the development of a number of teaching methods designed to engage students in the learning process. In this paper we will refer to these techniques as a whole as ‘active-student’ methods, while teaching methods which do not emphasize student engagement will be called ‘passive-student’ methods. This terminology is used simply to facilitate the discussion. We recognize that student activity or passivity is a matter of degree across all teaching methods and indeed varies widely from student to student. It is the purpose of the present study to compare the outcomes achieved by a teaching style based on these research-based active-student methods with those achieved by the lecturing style commonly employed in university physics departments.

Teaching methods

The specific active-student teaching methods used in this study were: Just-in-Time-Teaching (JiTT), Interactive Lecturing Demonstrations (ILD), Peer Instruction (PI) and the Personal Assistants for Learning (PAL) online tutoring programme. This approach was compared to a standard lecture style in a controlled study. The protocols for each of these teaching methods are discussed below.

Just-in-Time-Teaching (Novak et al. 1999) is a teaching method in which student understanding is assessed prior to lecturing and the content of the lecture is adjusted in response to the assessment results. The JiTT method identifies the student strengths and weaknesses on the topic about to be presented so that the instructor is able to tailor the lecture to concentrate upon the identified weaknesses, and avoid material with which the students are already competent. Novak et al. (1999) trialled their technique in three different institutions: a university, an air force academy and a college. All of these trials met with notable success.

Peer Instruction (PI) was introduced by Mazur (1997) and is a teaching method in which students confer with one another during class time in pairs or small groups using carefully designed conceptual questions or ‘ConcepTests’ as the stimulus for discussion. These questions concentrate on strengthening students' understanding of the conceptual basis of physics rather than requiring the manipulation and application of standard formulae. A typical class format in this teaching method involves the presentation of a ConcepTest, after which students write their individual answers to the multiple-choice question. Groups then form and each student endeavours to convince their partner(s) that their own answer is correct. If necessary, each student then records a revised answer. The instructor asks for a show of hands to each possible answer and provides an additional explanation of the correct answer. The ‘show of hands’ response system is rapidly being replaced by electronic response systems, i.e. ‘clickers’. These were not available at the time at which this study was conducted. Again this technique allows explanations to be tailored to the level of understanding in the class by allowing the students themselves to provide a substantial portion of the teaching.

Interactive Lecture Demonstrations (ILD) have been highlighted by Sokoloff and Thornton (1997) as an effective teaching method. Their technique is similar to Mazur's (1997) but uses lecture demonstrations instead of ConcepTests. Initially the instructor explains the demonstration which will take place and asks the students to write a prediction about its outcome. After the prediction sheets are collected, the students form small groups and discuss their ideas. Only when each student has recorded their new prediction does the instructor draw the students into discussing the most common predictions from the class. At this point the instructor performs the demonstration and invites student discussion of the outcome. Finally, the instructor describes and explains analogous physical situations which differ in their surface ‘appearance’ but which have the same underlying concept as the demonstration.

Reif and Scott's (1999) ‘Personal Assistants for Learning System’ (PALS) is an online tutoring tool which attempts to improve problem-solving skills. The design of the system is based upon detailed analysis of the cognitive functions students require in order to solve physics problems (Reif and Scott 1999). The PALS presents a carefully selected group of physics problems and guides the student through the problem-solving process. This process is directed by the tutor in a way that encourages the learning and adoption of those cognitive strategies which are commonly used by expert problem-solvers (i.e. experienced physicists). Reif and Scott used a reciprocal-teaching method which involved three different types of PALS tutorial: (i) PALS coaches the student (who practises implementing the actions as directed by the computer programme), (ii) the student coaches the PALS (practising making appropriate decisions and correcting any errors the computer programme makes), and (iii) independent practice (the student works independently through a problem and the computer assesses their performance). In this way the PALS gives the student practice at each step of problem-solving: deciding, implementing and assessing. The PALS also keeps track of a student's understanding as they progress, and if the student displays any difficulty with a particular concept, the PALS will direct the student to a ‘Knowledge Bank’, a very brief text which provides the basic factual information required for the problem currently being attempted.

Purpose of the study

We have conducted a study designed to investigate the effectiveness of active-student and passive-student teaching methods in a very short course on Newtonian mechanics. The aim of this study was to perform a controlled comparison of these two teaching methods and to determine the speed with which any difference in learning gain between these two methods appears. As such, this study rests upon the direct controlled comparison of passive-student and active-student teaching methods. Two different groups of high-school students were taught the same material by the same instructor, using the two different teaching protocols. To our knowledge such a direct comparison is not represented elsewhere in the literature.

Additionally, this study took place over a 2-week period, and involved a maximum of 4 hours contact time with the students. We were thus looking for the very early onset of learning gains. Finally, we included a survey of student satisfaction in our assessment programme, and report on student perceptions of the two courses. As an ancillary study, we collected data concerning sex differences in learning gains and these data will also be presented.

Student sample

Sixty-seven students from 10 local schools took part in the study. Five schools were allocated to the active-student group (school deciles range 6–10, mean 8.4) and five to the passive-student group (school deciles range 6–10, mean 8.0). The ages of these students varied from 15 to 18 years and all were in their 12th or 13th year of schooling. Of the 31 students who completed the active-student course, 17 (55%) were female and 14 (45%) male; 15 (48%) were completing their final year at secondary school (including physics) and 16 (52%) were completing their second to last year in secondary school (including physics). Their mean ratings of their abilities were physics 2.5, mathematics 2.3, other science 2.6, and other academic 2.5 from a 5-point scale where 1 indicated that the student viewed themselves as ‘very good’ in that area and 5 indicated that the student viewed themselves as ‘very poor’ in that area.

The 36 students who completed the passive-student group consisted of 17 (47%) females and 19 (53%) males. Eighteen (50%) were completing their final year at secondary school (including physics) and 18 (50%) were completing their second to last year in secondary school (including physics). Their mean ratings of their abilities were physics 2.9, mathematics 2.3, other science 2.4, and other academic 2.6.

All students taking part in the study had studied Newtonian mechanics within the last 18 months, at a level appropriate to their age group. To provide an incentive for students to attend lectures, do the homework etc. (and to reward them for their considerable time and effort) students were paid NZ$100 if they completed all components of the course. Not all of the students who were originally invited to take part in the study completed the programme. Some students failed to complete the study due to external commitments, and two students were asked to leave the study after failing to complete the required homework by the due date. A number of those students originally invited to take part in the study did not accept this invitation. A comparison of students (28) who either failed to complete the study or declined the initial invitation to take part in the study with those who accepted and did complete the programme (67) showed no statistically significant differences in terms of age, school decile and self-reported academic ability.

Local secondary-school students were invited to apply to take part in the study. All applicants were asked to provide information such as age, sex and at what level in the NCEA framework they had achieved in physics.1 Students were also asked to rate their own abilities in mathematics, physics, science and general academic ability.

The two study groups were made as similar as possible using dynamic randomization to allocate schools to the active-student teaching program. The two groups were thus as similar as possible in terms of sex and NCEA level categories, and mean self-reported physics ability. Randomization was performed at the school level to avoid contamination of students within the schools, i.e. to reduce the likelihood of collaboration between the study groups at school.

Statistical methods

Students in both groups were pre-tested and post-tested using the Force Concept Inventory (FCI) (Hestenes et al. 1992; Hestenes & Halloun 1995) and post tested using the Mechanics Baseline Test (MBT) (Hestenes, D. and Wells, M. 1992). The pre-test/post-test design allowed us to construct various measures of learning gain. These assessment instruments, and the learning gain measures constructed using them, will be discussed in more detail in a later section. Students were also asked to fill in an attitude survey at the end of the study. The data collected from these assessment instruments were subjected to careful statistical analysis.

Models were constructed that compared learning gains or final FCI or MBT scores while controlling for baseline performance in the case of FCI, and sex and NCEA level in all models. Clustering of students within schools was accounted for in all cases. Interactions between group and NCEA level or sex were retained in the models where P < 0.20. Analyses were performed using SAS 9.1.2 (SAS Institute Inc., Cary, NC, USA). P<0.05 was considered statistically significant.

Teaching protocols

The study covered mechanics at a level which would be appropriate in a standard first-year university introductory physics course. The courses covered introductory kinematics and some material on Newton's Laws over the 2-week duration of the study.

Neither teaching method included laboratory sessions or tutorials. The study was designed to compare the effectiveness of the two different styles of lecturing and the inclusion of tutorial or laboratory teaching would have introduced unnecessary confounding factors. Both groups were taught by the same lecturer and students were asked not to approach the lecturer outside class time.

Passive-student teaching protocol

The passive-student teaching protocol was based on the methods used in standard first-year introductory university physics courses. Students attended two 1-hour lectures (attendance was recorded), and completed a single homework assignment each week. The instructor was asked to adopt an ‘oratory’ style, standing at the front of the lecture hall discussing the notes as presented in a set of overheads. The pace of delivery was determined by the lecturer. Any questions volunteered by students during the hour were answered. At the beginning of the first lecture, students were given the entire set of overheads in note form.

Weekly homework was accessed on-line via a standard course management programme and students were required to submit their homework on time in order to be paid for their participation in the study. The homework consisted of both conceptual and quantitative questions and, with the exception of one question, had multiple-choice answers. Both groups were given the same homework assignments.

Students were informed that the photocopied sections of the textbook (Knight 2004) were recommended reading for the course; however, reading these sections was not compulsory and no reading tests were run. No incentives to read the textbook were given apart from the standard exhortation that it would be very useful. This approach to course reading was taken since requiring course reading was considered to be a central component of active-student teaching methods. Results of a student survey showed that approximately one-third of students read the textbook at some time during the course.

Active-student teaching protocol

The methods used in the active-student programme were a combination of Just-in-Time-Teaching (JITT), Peer Instruction, online tutoring using the PALS, and ensuring that students completed the assigned reading.

The active-student teaching protocol was textbook-based in that pre-reading from an assigned textbook was assumed. Students were assigned sections of the textbook to read and work through before the class, so that when students come to the lecture they are already acquainted with the days’ lecture material. By identifying the elements of the reading that the students have not fully grasped (see JiTT test below) the lecturer was able to focus the lecture in on the more difficult sections of the reading.

Students were issued with a course book, the appropriate photocopied sections of the latest (at the time) research-based textbook by Randall Knight (Knight 2004). This textbook acts as a teaching guide rather than the conventional ‘physics encyclopedia’ and is well-suited for pre-class reading.

We presumed the students were unlikely to do the assigned reading without an incentive, so we required that they complete a short on-line reading test before attending the class. The questions were designed to ensure the students had read the textbook. An example reading test question is:

Forces originating in the environment are called

(a) environmental forces

(b) internal forces

(c) external forces

(d) we can ignore forces originating in the environment

The phrase ‘Forces originating in the environment are called external forces’ appeared in the ‘Newton's Third Law’ section of the textbook.

Students were required to get 100% in the reading test, but could repeat the test as many times as needed. The reading test closed 6 hours before the lecture.

Just-in-Time-Teaching tests were used after the reading test to identify those elements of the reading that the students had not fully grasped. These were approximately 25 minutes long and, like the reading test, were due 6 hours before the lecture. Students were not required to achieve 100% in this test, and were only able to sit each JiTT test once.

The following is an example of one of the tests given during the study.

Car B is stopped for a red light. Car A, which has the twice the mass of car B, doesn't see the red light and runs into the back of B. Which of the following statements is true?

(a) B exerts a force on A but A doesn't exert a force on B

(b) B exerts a larger force on A than A exerts on B.

(c) B exerts the same amount of force on A as A exerts on B.

(d) A exerts a larger force on B than B exerts on A

(e) A exerts a force on B but B doesn't exert a force on A.

Before each lecture a summary of student responses to these questions was compiled. Using this information, lectures were constructed to address specific topics which the students were finding troublesome. Thus no lecture time was spent covering topics the students already understood, moreover, topics covered in the lecture were those that the majority of the class had difficulty with, in an attempt to make the lectures valuable to all students.

There was a single lecture session each week in the active-student teaching protocol. This lecture was given by the same lecturer and held in the same room as the passive-student programme. Each lecture consisted of approximately six short ‘presentations’ of important concepts that had been identified as requiring attention in the JiTT test held before the lecture. These presentations included a ‘mini-lecture’, often an interactive demonstration and a ConcepTest. Students were issued a set of lecture notes at the beginning of the first lecture. These notes were largely blank, except for concept headings, introductions to interactive demonstrations and problems which the lecturer intended to work through in class.

A short (5–8 minute) lecture was given on each concept. This ‘lecture’ usually included a worked example, with student input into the problem-solving process. For example, while drawing a force diagram for a free-fall problem the lecturer asked students to indicate, by a show of hands, whether the acceleration vector should point ‘up’ or ‘down’.

At the end of the mini-lecture students were presented with a concept test. These concept tests were multiple-choice questions that probe students’ understanding of the topic just presented. Where possible, the incorrect answer choices for each question were based on some of the common student misconceptions which have been identified in the research literature.

Lecture demonstrations were used in combination with the concept tests discussed above. Each demonstration was introduced by the lecturer and the students were invited to take notes or fill in a ‘Method’ section which had been included with the course notes. The students were asked to predict the outcome of the demonstration, indicating their prediction by a show of hands. The demonstration then proceeded and the outcome of the demonstration was used to initiate class discussion of the physics involved. In one such demonstration, a stream of milk droplets were directed into a bucket and illuminated with a strobe light. Students observed the stream of droplets with the classroom lights on. They were told that the droplets were released from a flask at a constant rate and were told how the strobe light worked. They were then asked to predict what they would see when the classroom lights were turned off and the strobe light was turned on. Students then compared their predictions with observation when the class room lights were turned off. The students observed that the spacing of the droplets was much larger at the bottom of the stream than at the top, as most of them had predicted.

Such predictions were not always required. In some cases students were simply asked to watch a demonstration, think quantitatively about what they had observed, and discuss the demonstration with the rest of the class and the lecturer.

Personal Assistants to Learning (PALS) are interactive, on-line tutorial programs which employ a ‘reciprocal teaching’ strategy. There are three distinct types of PAL tutorial:

i.	the computer coaches the student who practises implementing actions as instructed;
ii.	the student coaches the computer, thus practising making appropriate problem-solving decisions;
iii.	the student works independently through a problem and the computer assesses their performance.

The PALS programme tracks student performance and is able to adjust the progress of the tutorial to accommodate variations in student understanding. The PALS programme includes a ‘knowledge base’ which is essentially a short online textbook, and directs students to the appropriate section of the knowledge base if required.

The homework for the active-student group was identical to that for the passive-student group. One homework assignment was set each week and students gained access to the homework by completing a minimum of four PALS tutorials, at which time a link to the homework appeared on their Blackboard course page.2

Students also had access to a number of ‘Physlets’, interactive computer simulations depicting physical phenomena. The Physlets were modelled on those by Christian and Belloni (2001) and were included in the programme as an optional activity for students. At the conclusion of the study we found that only three students had used the Physlets more than once during the course of the study.

The same lecturer presented both teaching programmes. While this lecturer regularly receives very good results in surveys of student satisfaction in her classes, she had not previously presented an active-student course

Assessment instruments

Two assessment instruments were used to assess the extent to which student understanding of the key concepts of Newtonian dynamics developed over the course of the study. These were the Force Concept Inventory (FCI) (Hestenes et al. 1992, Hestenes & Halloun 1995) and the Mechanics Baseline Test (MBT) ( Hestenes & Wells 1992). The FCI and the MBT were introduced by Hestenes, Wells and Swackhamer in 1992. The FCI is a 30-question, multiple-choice test designed to assess students’ conceptual understanding of Newtonian mechanics. Students are normally given 40 minutes to complete the test. The FCI probes students’ common-sense beliefs, identifies student misconceptions and is a useful measure of the degree to which an individual student or a class display a Newtonian conception of force. The test covers forces in one and two dimensions (including Newton's third law). A few kinematics questions relating to motion diagrams are also included. The Mechanics Baseline test is a 26-item, multiple-choice test designed to assess student competence at connecting mechanics concepts with problem-solving. The MBT is generally considered to be a slightly harder test than the FCI. Students were pre-tested using the FCI just prior to the beginning of the study and were post-tested with the FCI and the MBT.

Learning gains

The measures of learning used in this study are based on the difference between pre- and post-test scores in the FCI, in the form of either normalized learning gains, modified normalized learning gains, or as simple differences in score. The normalized learning gain factor (g factor), g (Hake 1998) is thus defined as

The normalized learning gain may be understood as the proportion of what was not known prior to the teaching programme that has been learnt (i.e. gained) over the course of that teaching programme.

There are broadly speaking two commonly accepted ways of reporting learning gain in the physics education literature: the average normalized learning gain and the individual normalized learning gain. The average learning gain is obtained by finding the difference of the class average of the pre- and post-test scores, and normalizing this difference with the difference between the class average pre-test score and 100%. The individual normalized learning gain calculates the normalized learning gain for each individual student; this may or may not then be averaged. These two different measures are useful in different circumstances. In our case the individual normalized learning gain was the most appropriate metric as our goal in this analysis was a comparison between only two classes.

Problems were encountered with the learning gain as defined above in dealing with one particular individual. This individual scored 29 out of 30 in the FCI pre-test, but only 27 out of 30 in the post-test, giving an individual normalized learning gain of −200%. The learning gains of all other students were positive, and it is reasonable to assume that the student in question obtained a score of 29 in the pre-test due to fortuitous guessing or insecure understanding on at least two questions, and was not so successful in attempting these questions in the post-test. However these scores were achieved it was clear that the loss of two marks in the post-test by this student would skew the entire analysis unless some modified version of the normalized learning gain was employed. Note that if a student with a pre-test score of 10 out of 30 had achieved only 8 out of 30 in the post-test, they would have produced an individual normalized learning gain of −10%.

Specifically to accommodate this issue we used a modified version of the normalized gain. In this metric, learning loss was defined in a way equivalent to learning gain, namely the proportion of the total score that could be lost which actually was lost. Learning losses are then defined as negative learning gains, and the individual and average learning gains calculated as usual. The normalized learning loss is thus defined in a way which corresponds to the normalized learning gain discussed above, as follows:

The normalized learning loss is only used where the post-test score is less than the pre-test score, is thus always negative, and is treated as a negative learning gain. The two measures, g and l, are added together to give a measure of learning gain which we will call the modified learning gain (note that a similar measure of learning loss was introduced independently by Marx and Cummings (1998). This measure may be positive or negative and reflects both learning gain and loss. Thus the student in question is assigned a learning loss of about 7% (i.e. a learning gain of −7%) and this is included in the statistical analysis reported below.

Raw FCI score differences, individual normalized learning gains and modified learning gains were calculated for both student groups. Analysis of these data presented a number of challenges which were a result both of the measures used and of the nature of the current study. Due to the extremely short teaching period used in this study, learning gains are small, as would be expected. This means that the choice of learning gain metric becomes critical, as will be seen in the following analysis of the FCI and MBT data.

Results

In terms of change in raw FCI score, while controlling for pre-test score, we find that group is a predictor of change in FCI score, with FCI scores being higher in the active-student teaching methods group by 1.92 marks (95% CI 0.39–3.45, P = 0.020). Interestingly, sex was also a significant determinant of change in FCI score, independently of the teaching protocol employed, with males scoring 1.80 marks higher than females (95% CI 0.21–3.40, P = 0.031). There was no evidence that school level affected change (the year 13 students were estimated to score 0.32 marks higher than the year 12 students, with a 95% confidence interval of −1.32–1.97, P = 0.662). There was no evidence of an interaction between sex and group (P = 0.149).

In considering FCI changes when measured using the standard individual normalized learning gain, the data are not nearly as clear. In terms of gain there was no statistically significant difference between the two teaching methods (we estimate that the learning gain was 0.1, or 10%, higher for the active-student group, 95% CI −0.04–0.24, but with a P-value of P = 0.136). There was a tendency for males to perform better than females using the standard gain score (males 0.22, or 22%, higher, 95% CI −0.05–0.49, P = 0.097). There was no evidence that school level affected gain; the year 13 students had an average gain that we estimated to be 0.03 lower than that of the year 12 students, 95% CI −0.13–0.20, but again with a P-value (P = 0.653) which indicated that this effect was not statistically significant.

As discussed above, the standard normalizing procedure badly skews the data due to the lower post-test score of one particularly good student. There are a number of analysis options for dealing with this issue. We could simply remove this student from the study, and attempt to justify this procedure in terms of some outlier analysis. This changes the analysis and results in a difference between the gains achieved by the different teaching groups with P = 0.042. However, this procedure would require some justification based on the experimental design or some special consideration relating to this datum point alone. We see no justification for such a procedure; students who score more highly in a pre-test than in a post-test seem to be a perfectly normal and expected outcome of education research. On the contrary, it appears that the normalization procedure itself is allowing this one student to exert considerable influence over the outcome of the model.

Rather than removing such students from the data, we suggest instead the use of the simple generalization of the normalized learning gain described above. This provides a measure of learning gain which takes into account the possibility of learning loss. When our data are analysed using the modified normalized learning gain defined above we find that the difference between the learning gains of the two groups is very nearly statistically significant, P = 0.056. We feel that this result is considerably more robust as it is found without excluding any students unnecessarily. Thus modification of the metric used to analyse our data is a vital consideration in determining the success or failure of the teaching methods studied.

At the conclusion of the study students were asked to repeat the FCI, and also to complete the Mechanics Baseline Test (MBT). The MBT is generally considered to be a slightly harder test than the FCI, and is designed to focus more on problem-solving skill rather than conceptual understanding. Given the efforts made at the beginning of the study to ensure that the two student cohorts were as alike as possible it is reasonable to assume that any differences in MBT score between the two groups are due to the differences in teaching method. A model predicting MBT scores indicated that the teaching method used was statistically significant in determining post-test MBT score, with scores 1.34 points higher (out of 26, i.e. 5.15%) in the passive-student group (95% CI 0.02–2.66, P = 0.048). School level was also predictive of MBT score with scores 1.75 points (6.73%) higher in the year 13 group (95% CI 0.30–3.20, P = 0.024). Interestingly, there was strong evidence of females benefiting more from the active-student program than males with regard to MBT score (program-by-sex interaction P = 0.009). There was evidence that males in the passive-student program scored higher than males in the active-student program by 1.73 points (6.65%) (95% CI 0.35–3.12, P = 0.020); while females in the active-student program scored higher than females in the passive-student program by 4.41 points (16.96%) (95% CI 1.21–7.61, P = 0.013). Within the active-student programme, females scored higher than males by 6.15 points (23.65%) (95% CI 1.98–10.31, P = 0.009). There was no evidence of a overall difference in post-test scores between males and females across the two groups (sex P = 0.211).

On the basis of these data it would appear that active-student teaching methods do not enhance problem-solving ability among male students but do enhance this ability among female students. However, these results are suggestive only and need to be confirmed in other studies.

Finally, attitude surveys were taken at the conclusion of the study; the purpose of these surveys was to determine student perception of the two teaching methods. Two different surveys were used due to the differences between the two teaching methods. The surveys asked the students to rate statements from 1 to 5 depending on whether they strongly agreed (1) or strongly disagreed (5) with the proposed statement. Students in the active-student programme were asked to rate 55 statements and students in the passive-student programme were asked to rate 35 statements. The extra 20 statements in the active-student survey polled student response to the new elements in this programme, such as physlets and the PALS software. Both surveys concluded with several open-ended questions where students were able to write comments regarding aspects of the course. The analysis of these surveys compared questions which were shared between the two surveys relating to general attitudes toward lecturing and physics.

The surveys showed that the active-student programme was considered to be more enjoyable (P = 0.022); students felt they learnt more in the active-student programme (P = 0.002), and also found lectures less boring in the active-student programme (P < 0.001).

Discussion

Several conclusions may be drawn from this study. Firstly, the analysis of pre- and post-test data using score difference as a measure of learning is fraught. These measures should be treated with considerable caution as the statistical significance of findings based upon these measures appears to be dependent on the particular form of learning gain used. Having said this, we are cautiously confident in saying that the active-student teaching methods trialled here were more effective in improving scores in the FCI than the passive-student teaching methods. This result indicates that this learning gain appears very early in the active-student teaching protocol. Further study is required to determine whether this rate of improvement is maintained or changes over the course of a longer active-student programme.

Secondly, it appears that while the active-student teaching methods improved student performance in the FCI more that the passive-student methods, the data concerning MBT scores are more complex. It appears that the MBT data indicate a marked difference between male and female participants. While male students improved their MBT score in the passive-student group, female students improved their MBT score in the active-student group. Furthermore the improvement in female MBT score in the active-student group was very large compared to the male improvement in the passive-student group. This result is particularly interesting as the active-student teaching method could be expected to improve FCI score as it is oriented towards the conceptual basis of mechanics, whereas the MBT tests for the application of these concepts in problem-solving. Thus it would appear that female students may be much more likely to connect the concepts of Newtonian mechanics to problem-solving techniques than male students, and they do this very early on in an active-student programme. Very similar results were found by Lorenzo et al. (2006). A recent paper by Miyake et al. (2010) suggests a possible cause of this effect. Miyake et al. find that a simple ‘values affirmation’ exercise improves the exam scores of female students from an average C grade to an average B grade. The study involved a simple intervention in which students were asked to write briefly about their core values (the ‘values affirmation’ exercise). This exercise has been shown to counteract the psychological effects of a difficult and stressful programme (such as physics or mathematics) on groups which are subject to negative stereotyping in these programmes.

Finally, regardless of the differences in learning gains in the FCI and student performance in the MBT, students clearly enjoyed the active-student teaching style considerably more than the passive-student teaching style. In the absence of any clear reduction in student learning, this fact alone is enough to recommend the active-student teaching methods trialled here over the passive-student teaching methods.

Notes

¹The NCEA (National Certificate of Educational Achievement) is New Zealand's main national qualification for school students from year 11 to year 13 and, among other things, is used to determine eligibility for entrance into New Zealand universities. NCEA is a relatively new qualification framework; its introduction was completed in 2003. NCEA uses standards-based assessment and has extensive internal and external assessment components. More information concerning the NCEA framework may be found at: http://www.nzqa.govt.nz/ncea/about/index.html

²The Blackboard Learning System is a course management programme produced by Blackboard Inc. (www.blackboard.com).