Score Reliability of Adolescent Alcohol Screening Measures: A Meta-Analytic Inquiry

ABSTRACT

This study describes the reliability reporting practices in empirical studies using eight adolescent alcohol screening tools and characterizes and explores variability in internal consistency estimates across samples. Of 119 observed administrations of these instruments, 40 (34%) reported usable reliability information. The Personal Experience Screening Questionnaire—Problem Severity scale generated average reliability estimates exceeding 0.90 (95% CI = 0.90–0.96) and the Adolescent Alcohol Involvement Scale generated average score reliability estimates below 0.80 (95% CI = 0.67–0.85). Average reliability estimates of the remaining instruments were distributed between these extremes. Sample characteristics were identified as potentially important predictors of variability in the reliability estimates of all the instruments and all instruments under evaluation generated more reliable scores in clinical settings (M = 0.89) as opposed to nonclinical settings (M = 0.82; r effect size (38) = 0.29, p < .10). Clinicians facing instrument selection decisions can use these data to guide their choices and researchers evaluating the performance of these instruments can use these data to inform their future studies.

KEYWORDS:

• Adolescent alcohol screening
• psychometrics
• reliability generalization
• reliability

This research was supported by grants from the National Institute on Alcohol Abuse and Alcoholism (R21-AA13423-01; Kim Wallace and Alan Shields) and the National Institute on Drug Abuse (DA-00326 and DA-15831; Roger Weiss). The authors thank Jennifer Gottlieb and Carolee McCall for their project development assistance and Nabil Haddad, PhD, Department of Psychology Chairperson at the University of Montana, for his support of this research.

Notes

Note. ^a Total number of studies using the identified instrument (all obtained from the peer-reviewed literature).

Note. K = the total number of independent samples; N = the number of participants within those samples. The AAIS total N of 11 samples yielded 2,501 respondents (while 12 samples provided reliability coefficients, only 11 specified the sample size).

^a Heterogeneity tests demonstrated that the unweighted weighted mean reliability coefficients for the AAIS were statistically lower then the other five measures as a set.

^†Heterogeneity tests demonstrated that the unweighted weighted mean reliability coefficients for the AAIs were statistically lower than the other five measures as a set.

Note. K = the total number of independent samples.

Note. Sample type was coded as clinical (1) and non-clinical (0).

∗Pearson r refers to the relationship between the unweighted reliability estimates and identified predictor variables.

∗∗Note: insufficient reporting of SD precluded the individual evaluation of the PESQ-PS, RAPI, DUSI-SUS, and YAAPST.