An exploration into the criteria used in assessing design activities with adaptive comparative judgment in technology education

Abstract

The use of design assignments for teaching, learning, and assessment is considered a signature of technology education. However, there are difficulties in the valid and reliable assessment of features of quality within designerly outputs. In light of recent educational reforms in Ireland, which see the introduction of classroom-based assessments centring on design in the technology subjects, it is paramount that the implementation of design assessment is critically considered. An exploratory study was conducted with a first year cohort of initial technology teacher education students (N = 126) which involved them completing a design assignment and subsequent assessment process through the use of adaptive comparative judgement (ACJ). In considering the use of ACJ as a potential tool for design assessment at post-primary level, data analysis focused on criteria used for assessment. Results indicate that quantitative variables, i.e. the amount of work done, can significantly predict performance (R² = .333, p < .001), however qualitative findings suggest that quantity may simply align with quality. Further results illustrate a significant yet practically meaningless bias may exist in the judgement of work through ACJ (ϕ = .082, p < .01) and that there was need to use varying criteria in the assessment of design outputs.

Keywords:

• adaptive comparative judgement
• assessment
• design
• educational reform
• technology education

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Jeffrey Buckley

Dr Jeffrey Buckley received his PhD from KTH Royal Institute of Technology, Sweden, in the area of spatial ability and learning in technology education. He is a qualified post-primary teacher of Design and Communication Graphics and Construction Studies. As a member of the Technology Education Research Group (TERG) he is now a post-doctoral researcher in engineering education in Athlone Institute of Technology, Ireland, and KTH Royal Institute of Technology, Sweden. His main research interest is in how people learn. He is particularly interested in how cognitive abilities such as spatial ability effect students’ capacity to learn, and how levels of prior knowledge impact on further learning. Jeffrey is also interested in inclusivity in engineering and technology education, particularly in relation to stereotypes and misconceptions that people may have about technical subject areas. Jeffrey is currently actively involved in a number of national and international research projects in the areas of learning and educational diversity, and is an active post-graduate supervisor in these areas.

Donal Canty

Dr Donal Canty is a qualified Post-Primary teacher with 8 years’ classroom experience. Donal’s research interests are in the areas of pedagogy and assessment. His doctoral studies investigated the impact of holistic assessment using Adaptive Comparative Judgement (ACJ) on student learning and capability. Donal is one of the founding members of the Technology Education Research Group (TERG) and has led both national and international research projects. He has significant experience in managing school-based research that focus on assessment practices and technology mediated teaching and learning. Donal has extensive experience of programme design for both secondary and higher education and is currently a lecturer in the School of Education at the University of Limerick.

Niall Seery

Dr Niall Seery is currently the Vice President of Academic Affairs and Registrar at Athlone Institute of Technology. He is a qualified secondary school teacher of Engineering, Technology and Design and Communication Graphics. Niall has a background in Technology Teacher Education, where he spent 15 years as an academic at the University of Limerick with a specialist interest in pedagogical practice. He has served as director of studies at undergraduate and masters’ level for Technology Teacher Education, while also developing an emerging research agenda. Niall also served as a visiting Associate Professor of Technology Education at the Royal Institute of Technology, KTH in Stockholm. Niall founded and still directs the Technology Education Research Group (TERG), where he actively supervises research students and participates in international research projects. He remains committed to advocating for technology and engineering education research and supporting the development of associated policy and practice.