Assessing Item Content-Relevance in Sport Psychology Scale-Construction Research: Issues and Recommendations

Abstract

During the item-construction phase of scale development, expert judges frequently rate the degree of match between the content of test items and the objectives to be measured by the test. However, Messick (1989) contended that systematic attempts to document and assess the item ratings provided by expert judges are not commonplace in the literature, and this is particularly true in the field of sport psychology. In response to Messick's concerns, the purpose of this study was to illustrate in a sport psychology context how a modified version of a procedure described by Hambleton (1980) can be used to provide a systematic quantitative assessment of judges' item content-relevance ratings. An expert panel of 38 judges rated the degree of match between 16 cognitive worry items and 4 latent worry dimensions to be measured by a newly constructed sport-specific anxiety measure. Issues regarding the composition of the expert panel and methods used to assess and report the experts' ratings are discussed.