Evaluation of diagnostic tools that tertiary teachers can apply to profile their students’ conceptions

ABSTRACT

A multi-institution collaborative team of Australian chemistry education researchers, teaching a total of over 3000 first year chemistry students annually, has explored a tool for diagnosing students' prior conceptions as they enter tertiary chemistry courses. Five core topics were selected and clusters of diagnostic items were assembled linking related concepts in each topic together. An ordered multiple choice assessment strategy was adopted to enable provision of formative feedback to students through combination of the specific distractors that they chose. Concept items were either sourced from existing research instruments or developed by the project team. The outcome is a diagnostic tool consisting of five topic clusters of five concept items that has been delivered in large introductory chemistry classes at five Australian institutions. Statistical analysis of data has enabled exploration of the composition and validity of the instrument including a comparison between delivery of the complete 25 item instrument with subsets of five items, clustered by topic. This analysis revealed that most items retained their validity when delivered in small clusters. Tensions between the assembly, validation and delivery of diagnostic instruments for the purposes of acquiring robust psychometric research data versus their pragmatic use are considered in this study.

KEYWORDS:

• Concept diagnostic instrument
• general chemistry
• psychometric analysis
• ordered multiple choice

Acknowledgements

The views expressed in this activity do not necessarily reflect the views of the Australian Government Office for Learning and Teaching. The significant contributions of project team members Hayden Dickson, Aaron Micallef and Md Abdullah Al Mamun to the development of tools and resources is gratefully acknowledged. We thank the students involved in the study. We also acknowledge insights from exchanges and discussions with the authors (Lawrie) regarding the application, analysis, validation of inventories including access to their own concept inventories by Jazilah Othman, Stephen Krause, Jennifer Lewis, Thomas Pentecost and Stacey Lowery-Bretz.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

Support for this activity has been provided by the Australian Government Office for Learning and Teaching (ID12-2277).