A Hierarchical Rater Model for Longitudinal Data

ABSTRACT

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.

KEYWORDS:

• Autoregressive
• hierarchical rater model
• item response theory
• latent trait estimation
• longitudinal
• ratings
• time series
• trends

Article Information

Conflict of Interest Disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical Principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was supported by Grant SES 1324587 from the National Science Foundation.

Role of the Funders/Sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: This work was supported by Grant SES 1324587 from the National Science Foundation. We are grateful to the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing high performance computing resources that have contributed to the research results reported within this paper (URL: http://www.tacc.utexas.edu). We are especially grateful to David Walling, Ruizhu Huang, and Lei Huang for their personal assistance with implementing parallelization on the supercomputer, Stampede. This work also used resources from the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562. Special thanks to Yisi Wang (former student at UT Austin) for her data analysis work early in the project. We also would like to gratefully acknowledge Angela Duckworth for granting permission to use data from the Character Development in Adolescence Project (CDAP; https://coa.stanford.edu/content/character-development-adolescence). The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions or the National Science Foundation is not intended and should not be inferred.

Notes

1 It should be noted that the GPCM is appropriate for use with rating scale data despite traditional viewpoints on how the GPCM should be used (Hambleton, van der Linden, & Wells, 2010, p. 31).

2 Equation (4(4) ) presents the Normal hyperparameters in terms of variances, but when discussing the actual values for the hyperparameters here, we provide SDs to assist the readers’ understanding.

3 For more information on Stampede, visit: https://www.tacc.utexas.edu/stampede/

4 In many practical situations, the analyst may request more iterations on the same set of chains to arrive at convergence, however, the remote manner in which we ran our simulation study did not support this process. For this reason, we simply started a new set of chains with a new set of randomly generated initial values.

5 Figure A2 in the Supplementary Material provides a pictorial depiction of this phenomenon.