Generalizability of Ratings of Motor Performances with Reference to Various Observational Designs

Abstract

The purpose of this paper is to examine the appropriateness of several generalizability coefficients which might be computed from observational data based on a number of subjects, each being rated by the same observers over a number of trials. The extent to which various observational designs can impair the computation of some coefficients and/or alter their value is also examined. Generalization over raters and/or trials has led to the presentation of four different G-coefficient formulae. Considering trials as nested within subjects will not meaningfully restrict the calculation of the various coefficients of interest but will yield lower values for most of them whenever there is a substantial trial effect. Considering trials as nested within subjects and within raters will, on the other hand, restrict considerably the computation of the G-coefficients of interest, and will also yield lower coefficients whenever there is a substantial trial effect. Cases in which subjects are observed by different observers are briefly examined, and should be avoided whenever a generalizability analysis is intended.

Key Words:

• generalizability
• inter-rater
• inter-trial
• motor performance
• observational measurement
• ratings
• reliability