The concept of validity in theory and practice

Abstract

The concept of validity, as described in the literature, has changed over time to become a broad and rather complex issue. The purpose of this paper is to investigate if practice has followed theory, or if there is a gap between validity in theory and validity in practice. It compares the theoretical development of the concept of validity with the methodology adopted in validity studies over time. Important phases in the history of validity, and also common arguments for and against traditional and modern validity perspectives, are presented and discussed. Thereafter, three Swedish research projects aiming to validate instruments used for selection to higher education are described. The idea is to use these projects as examples of contemporary practice, and to compare their designs, research questions and outcomes with how validity was theoretically described during their specific period of time. The conclusions from these comparisons are that practices seem to have followed theory when it comes to how the validity research programmes have been designed, but not when it comes to how they then were carried out in practice. This gap between theory and practice seems to have increased with the introduction of broader and more modern validity perspectives. The scope of the research is more extensive but results are fragmented and there is no evidence of a ‘unified’ validity argument, which has been one of the central aspects in modern validity theory. This supports the arguments that validity theory is difficult to put into practice and that there is a need for guidance on how to prioritise validity questions and interpret validity evidence.

Acknowledgements

The authors would like to thank the participants in the seventh annual AEA‐Europe conference in Naples for interesting comments and suggestions on the topic of this paper. The authors would also like to thank Professor Widar Henriksson, the anonymous reviewers, and the editor of the journal Assessment in Education: Principles, policy & practice for valuable comments on the content and structure of this paper.