Creating a test blueprint for a progress testing program: A paired-comparisons approach

Abstract

Context: Creating a new testing program requires the development of a test blueprint that will determine how the items on each test form are distributed across possible content areas and practice domains. To achieve validity, categories of a blueprint are typically based on the judgments of content experts. How experts judgments are elicited and combined is important to the quality of resulting test blueprints.

Methods: Content experts in dentistry participated in a day-long faculty-wide workshop to discuss, refine, and confirm the categories and their relative weights. After reaching agreement on categories and their definitions, experts judged the relative importance between category pairs, registering their judgments anonymously using iClicker, an audience response system. Judgments were combined in two ways: a simple calculation that could be performed during the workshop and a multidimensional scaling of the judgments performed later.

Results: Content experts were able to produce a set of relative weights using this approach. The multidimensional scaling yielded a three-dimensional model with the potential to provide deeper insights into the basis of the experts’ judgments.

Conclusion: The approach developed and demonstrated in this study can be applied across academic disciplines to elicit and combine content experts judgments for the development of test blueprints.

Acknowledgements

Dr. David Kohler, at that time of workshop, was the postdoc who helped to run the workshop.

HCvB developed the original idea on all design elements of the workshop. A postdoctoral fellow of HCvB suggested using iClicker System. RC suggested to use and conducted paired comparisons to further analyze the workshop results. Both authors drafted and revised the paper and approved this version for publication. With the ADEA/ADEAGies Foundation Education Fellowship 2016–2017, HCvB was able to focus on completing the first phase of the study.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Ethical approval

The data reported in this paper are anonymized. It is covered by the University of British Columbia Behavioral Research Ethics Board H16-00957.

Glossary

Progress testing: This type of testing practice usually is creating tests that sample the complete knowledge domains expected of the students upon completion of their professional program, regardless of the year level in which it is administered. Often one test with several equivalent forms is designed and is administered to all students in all years at the same time and at regular intervals. The resultant scores provide longitudinal, repeatedly measured, and curriculum-independent assessment of the knowledge objectives of the entire professional program.

Notes on contributors

HsingChi von Bergmann, PhD, is an Associate Professor, Education Research, Co-Chair of CTEC Assessment and Ed-Tech Subcommittee, Faculty of Dentistry, University of British Columbia.

Ruth A. Childs, PhD, is a Professor, Department of Leadership, Higher and Adult Education, University of Toronto.

Additional information

Funding

This research was funded by the UBC Teaching and Learning Enhancement Funds.