An online student testlet-generation system: design principles and evidence supporting its learning potential and usefulness

ABSTRACT

Currently, 50 + learning systems supporting student question-generation (SQG) activities have been developed. While generating questions of different types is supported in many of these systems, systems allowing students to generate questions around a scenario (i.e. student testlet-generation, STG) are not yet available. Noting the increasing prevalence of testlets in contemporary testing and their learning potential through the lens of anchored instruction, this study aimed to develop a versatile, scaffolded online STG learning system. The results from three evaluative studies supported the learning potential of the world’s first-ever STG system and the usefulness of its distinctive functionalities. Specifically, significantly more participants regarded the online STG task as promoting better learning, with the most prominent feature of STG – the scenario highlighted for the pronounced learning benefits. Additionally, given that the distinct affordances associated with STG based on given text and graphics were well-noted, the practicality of the ‘multi-presentation' design concept for scenarios was verified. Furthermore, data from the participants’ perceived usefulness of editable and self-created scenarios validated the importance of the ‘editability' and ‘self-expression' design concepts. Finally, with the distinctive functionalities incorporated in the developed STG system having their respective supporters, the concept of building a diversified STG learning space is upheld.

KEYWORDS:

• Diversified online learning systems
• learning potential
• scaffolding
• student-generated questions
• student-generated testlets

Acknowledgements

Thanks are extended to my research assistants, Ya-Hsin Chang, Zi-Yu Hsu, and Chih-Chung Lin for their assistance in collecting the evaluative data reported in this work. Thanks are also extended to computer engineers Lu-Chun Pan and Wei-Hao Du for their great assistance in constructing and testing the focal learning system. Special thanks to Yu-Cheng Chen for documenting the system's architecture, algorithms, and methodologies.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 For instance, testlets consisting of both selection (e.g., multiple-choice) and construction (e.g., essay, short-answer) question types are used in PISA to measure 15-year-old students’ scholastic performance on mathematics, science, and reading (Organization for Economic Co-operation and Development, 2018) whereas it is used in TIMSS to assess the mathematics and science achievement of students at grades 4 and 8 (TIMSS & PIRLS International Study Center, n.d.).

Additional information

Funding

This work was supported by the Ministry of Science and Technology, Taiwan [Grant Number MOST 108-2511-H-006-007-MY3].

Notes on contributors

Fu-Yun Yu

Fu-Yun Yu is Distinguished Professor at the Institute of Education of National Cheng Kung University, Taiwan. Her current research interests are innovative assessment strategies for teaching and learning, peer-assisted learning, and design of customizable online learning environments.