Evidence to support test interpretation: Evaluating the content validity of a new item set for the Transition Pragmatics Interview

Abstract

Purpose

There is little consensus on evidence-based practice guidelines for the selection of criterion-referenced assessments. Having confidence in scores from criterion-referenced assessments requires evidence that items align with their intended constructs. The purposes of these studies were to demonstrate evidence of content validity for the revised item set of a developing social communication assessment and to provide clinicians with a model of content validity evaluations that can be generalised to the review of other assessments.

Method

In Study 1, 10 experts rated 25 newly-developed items for how well they represented the intended construct. In Study 2, seven participants ages 14–20 were administered the Three Step Test Interview to assess their cognitive processes for responding to new items. Examinee responses were coded for construct-relevant and construct-irrelevant factors.

Result

Twenty-three of the 25 newly-developed items were deemed representative of the intended construct by experts and elicited construct-relevant response processes from examinees.

Conclusion

The integration of expert review and examinee cognitive interviewing provides a more complete evaluation of the alignment of the items to their intended construct. Transparent reports of the methods and findings of content validity studies strengthen the ability of clinicians to select criterion-referenced assessments that support valid decisions.

Keywords:

• adolescents
• psychometric properties
• content validity
• assessment
• social communication
• language

Acknowledgements

The content is the authors’ responsibility and does not necessarily represent the official views of the NIH.

Declaration of interest

No potential conflict of interest was reported by the authors.

Additional information

Funding

This research was supported by the National Institute on Deafness and Other Communication Disorders of the National Institutes of Health (NIH), Award R15DC020521.