The discriminative learning gain: a two-parameter quantification of the difference in learning success between courses

ABSTRACT

Pre- and post-tests are often used in Engineering Education Research to assess teaching and learning. Sometimes, it is reasonable or even necessary to use a different pre- than post-test. In that case, it is difficult to analyse the data with traditional methods such as average normalised gain or normalised change scores. We propose to use the so-called discriminative learning gain (DLG) to analyse such data. This two-parameter statistic describes the post-test performance of model students, taking into account varying pre-test performances. Thus, it describes not only the learning of the average student, but also the discriminative effect of the instruction with respect to initial performance. Teachers and researchers can use the DLG as a tool to effectively compare course performances. Using confidence bounds, the difference in learning success among courses can be quantified and easily visualised. Consequently, informed conclusions can be drawn if courses differ in effectiveness. This article demonstrates how to apply and interpret the DLG. Limitations of the method and the application to the special case of identical pre- and post-tests are discussed.

KEYWORDS:

• Learning gain
• course assessment method
• linear regression
• non-identical pre-/post-tests
• concept inventories (CIs)

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Although exams are often created by highly experienced instructors, they generally cannot provide the same standard of validity.

2. This is especially true for class averages. The chances of obtaining negative gains are higher if the measure is applied to individual scores.

3. if the following assumptions are met: for each pre-test score, the corresponding observations of the post-test scores are independent, normally distributed and have the same variance (Neter, Wasserman, and Kutner 1985, 123).

4. Note that the $\alpha$ in Equation (19) has the same meaning as in Equation (9) but their values can be chosen independently.

5. formerly known as Statics Concept Inventory.

6. Coe (2002, 4) presents multiple ways to interpret these numbers.

Additional information

Funding

This work was supported by the German Federal Ministry of Education and Research (BMBF) [grant number 01PL11047]. Any opinions expressed here are those of the authors.

Notes on contributors

Julie Direnga

Julie Direnga is a doctoral student and research assistant in the Engineering Education Research Group at Hamburg University of Technology (TUHH) and an engineer with degrees in General Engineering Science (B.Sc.) and Mechatronics (M.Sc.). Her current research focuses on the assessment of conceptual understanding in introductory mechanics courses.

Dion Timmermann

Dion Timmermann is an electrical engineer (M.Sc.) and educational researcher. He is currently pursuing a Ph.D. in university-level engineering education research at Hamburg University of Technology. His research focuses on students’ conceptual understanding of electrical engineering and the design of instructional materials fostering conceptual change.

Ferdinand Kieckhäfer

Ferdinand Kieckhäfer is a doctoral student and research assistant in the Engineering Education Research Group at Hamburg University of Technology with degrees in General Engineering Science (B.Sc.) and Electrical Engineering (M.Sc.). His areas of research are conceptual understanding in control engineering and the development of concept inventories. 

Andrea Brose

Andrea Brose is a mathematician (Ph.D.) and the head of the Center for Teaching and Learning at Hamburg University of Technology (TUHH), where she coordinates the professional development of faculty and teaching staff in engineering. Previously, she worked as a post-doc in the Engineering Education Research Group at TUHH, where she helped develop and evaluate of evidence-based teaching and learning materials.

Christian Kautz

Christian Kautz is the head of the Engineering Education Research Group at Hamburg University of Technology. A physicist by training, he holds a degree in university-level physics education research (Ph.D.) from the University of Washington. After three further years in the US, as an assistant professor in the Department of Physics at Syracuse University, he returned to Germany. Besides teaching seminars on engineering education and teaching various introductory engineering courses, he carries out research on student understanding of core concepts in these subjects, as well as on more general cognitive skills.