A Hierarchical Rater Model for Longitudinal Data

References

• Andersen, E. B. (1985). Estimating latent correlations between repeated testings. Psychometrika, 50(1), 3–16. https://doi.org/10.1007/BF02294143
   Web of Science ®Google Scholar
• Andrade, D. F., & Tavares, H. R. (2005). Item response theory for longitudinal data: population parameter estimation. Journal of Multivariate Analysis, 95(1), 1–22. https://doi.org/10.1016/j.jmva.2004.07.005
   Web of Science ®Google Scholar
• Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573. https://doi.org/10.1007/BF02293814
   Web of Science ®Google Scholar
• Azevedo, C. L., Fox, J. P., & Andrade, D. F. (2015). Longitudinal multiple-group IRT modelling: covariance pattern selection using MCMC and RJMCMC. International Journal of Quantitative Research in Education, 2, 213–243. https://doi.org/10.1504/IJQRE.2015.071737
   Google Scholar
• Box, G. E. P., Jenkins, G. M., & Reinsel, G. C. (2013). Time series analysis: Forecasting and control. Hoboken, NJ: John Wiley & Sons. doi: 10.1002/9781118619193
   Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag. doi: 10.1007/978-1-4757-3456-0
   Google Scholar
• Cagnone, S., Moustaki, I., & Vasdekis, V. (2009). Latent variable models for multivariate longitudinal ordinal responses. British Journal of Mathematical and Statistical Psychology, 62(2), 401–415. https://doi.org/10.1348/000711008 × 320134
   PubMed Web of Science ®Google Scholar
• Casabianca, J. M., Junker, B. W., & Patz, R. (2016). The hierarchical rater model. In W. J. van der Linden (Ed.), Handbook of modern item response theory (pp. 449–465). Boca Raton, FL: Chapman & Hall/CRC.
   Google Scholar
• Casabianca, J. M., Lockwood, J. R., & McCaffrey, D. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75, 311–337. https://doi.org/10.1177/0013164414539163
   PubMed Web of Science ®Google Scholar
• DeCarlo, L. T., Kim, Y., & Johnson, M. S. (2011). A hierarchical rater model for constructed responses, with a signal detection rater model. Journal of Educational Measurement, 48(3), 333–356. https://doi.org/10.1111/j.1745-3984.2011.00143.x
   Web of Science ®Google Scholar
• Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (Grit-S). Journal of Personality Assessment, 91, 166–174. https://doi.org/10.1080/00223890802634290
   PubMed Web of Science ®Google Scholar
• Dunson, D. B. (2003). Dynamic latent trait models for multidimensional longitudinal data. Journal of the American Statistical Association, 98(463), 555–563. https://doi.org/10.1198/016214503000000387
   Web of Science ®Google Scholar
• Eid, M., & Hoffmann, L. (1998). Measuring variability and change with an item response model for polytomous variables. Journal of Educational and Behavioral Statistics, 23(3), 193–215. https://doi.org/10.1198/016214503000000387
   Web of Science ®Google Scholar
• Embretson, S. E. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56(3), 495–515. https://doi.org/10.1007/BF02294487
   Web of Science ®Google Scholar
• Embretson, S. E. (1997). Structured ability models in tests designed from cognitive theory. In M. Wilson, G. Engelhard, Jr., & K. Draney (Eds.), Objective measurement: theory into practice (Vol. 4, pp. 223–236). Greenwich, CT: Ablex.
   Google Scholar
• Froh, J. J., Fan, J., Emmons, R. A., Bono, G., Huebner, E. S., & Watkins, P. (2011). Measuring gratitude in youth: Assessing the psychometric properties of adult gratitude scales in children and adolescents. Psychological Assessment, 23(2), 311–324. https://doi.org/10.1037/a0021590
   PubMed Web of Science ®Google Scholar
• Fu, Z. H., Tao, J., Shi, N. Z., Zhang, M., & Lin, N. (2011). Analyzing longitudinal item response data via the pairwise fitting method. Multivariate Behavioral Research, 46(4), 669–690. https://doi.org/10.1080/00273171.2011.589279
   PubMed Web of Science ®Google Scholar
• Gelman, A., Meng, X. L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6(4), 733–760. Retrieved from http://www.jstor.org/stable/24306036
   Web of Science ®Google Scholar
• Gelman, A., & Rubin, D. B. (1996). Markov chain Monte Carlo methods in biostatistics, Statistical Methods in Medical Research, 5(4), 339–355. https://doi.org/10.1177/096228029600500402
   PubMedGoogle Scholar
• Guo, S. (2014). Correction of rater effects in longitudinal research with a cross-classified random effects model, Applied Psychological Measurement, 38(1), 37–60. https://doi.org/10.1177/0146621613488821
   Web of Science ®Google Scholar
• Hambleton, R. K., van der Linden, W. J., & Wells, C. S. (2010). IRT models for the analysis of polytomously-scored data: Brief and selected history of model building advances. In R. Ostini & M. Nehring (Eds.), Handbook of polytomous item response theory models (pp. 21–42). London: Routledge Academic.
   Google Scholar
• Hamilton, J. D. (1994). Time series analysis. Princeton, NJ: Princeton University Press.
   Google Scholar
• Harik, P., Clauser, B. E., Grabovsky, I., Nungester, R. J., Swanson, D., & Nandakumar, R. (2009). An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement, 46(1), 43–58. https://doi.org/10.1111/j.1745-3984.2009.01068.x
   Web of Science ®Google Scholar
• Hershberger, S. L., Molenaar, P. C. M., & Corneal, S. (1996). A hierarchy of univariate and multivariate structural time series models. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 159–194). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
   Google Scholar
• Hill, H. C., & Grossman, P. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384. https://doi.org/10.17763/haer.83.2.d11511403715u376
   Web of Science ®Google Scholar
• Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research, 18, 117–144. https://doi.org/10.1080/03610739208253916
   PubMed Web of Science ®Google Scholar
• Hoyt, W. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods, 5, 64–86. https://doi.org/10.1037/1082-989X.5.1.64
   PubMed Web of Science ®Google Scholar
• Hoyt, W. T., & Kerns, M. D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403–424. https://doi.org/10.1037/1082-989X.4.4.403
   Web of Science ®Google Scholar
• Hung, L. F., & Wang, W. C. (2012). The generalized multilevel facets model for longitudinal data. Journal of Educational and Behavioral Statistics, 37(2), 231–255. https://doi.org/10.3102/1076998611402503
   Web of Science ®Google Scholar
• IES (Institute of Education Sciences). (2010). Efficacy of schoolwide programs to promote social and character development and reduce problem behavior in elementary school children. Retrieved from http://ies.ed.gov/ncer/pubs/20112001/index.asp
   Google Scholar
• Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Journal of Fluids Engineering, 82(1), 35–45. https://doi.org/10.1115/1.3662552
   Google Scholar
• Kane, T.J., & Staiger, D.O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Bill & Melinda Gates Foundation. Retrieved from ERIC database. (ED540962).
   Google Scholar
• Kingsbury, F. A. (1922). Analyzing ratings and training raters. Journal of Personnel Research, 1, 377–383.
   Google Scholar
• Leckie, G., & Baird, J. A. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399–418. https://doi.org/10.1111/j.1745-3984.2011.00152.x
   Web of Science ®Google Scholar
• Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago, IL: MESA Press.
   Google Scholar
• Little, T. D. (2013). Longitudinal structural equation modeling. New York: Guilford Press.
   Google Scholar
• Liu, Y., Millsap, R. E., West, S. G., Tein, J.-Y., Tanaka, R., & Grimm, K. J. (2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. https://doi.org/10.1037/met0000075
   PubMed Web of Science ®Google Scholar
• Mariano, L. T. (2002). Information accumulation, model selection and rater behavior in constructed response student assessments (Doctoral dissertation). Carnegie Mellon University.
   Google Scholar
• Mariano, L. T., & Junker, B. W. (2007). Covariates of the rating process in hierarchical models for multiple ratings of test items. Journal of Educational and Behavioral Statistics, 32, 287–314. https://doi.org/10.3102/1076998606298033
   Web of Science ®Google Scholar
• McArdle, J. J., Petway, K. T., & Hishinuma, E. S. (2015). IRT for growth and change. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 435–456). New York: Routledge.
   Google Scholar
• Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543. https://doi.org/10.1007/BF02294825
   Web of Science ®Google Scholar
• Millsap, R. (2010). Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives, 4, 5–9. https://doi.org/10.1111/j.1750-8606.2009.00109.x
   Web of Science ®Google Scholar
• Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–177. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
   Web of Science ®Google Scholar
• Myford, C. M., & Wolfe, E. W. (2009). Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46(4), 371–389. https://doi.org/10.1111/j.1745-3984.2009.00088.x
   Web of Science ®Google Scholar
• Patz, R. J. (1996). Markov chain Monte Carlo methods for item response theory models with applications for the National Assessment of Educational Progress (Doctoral dissertation). Carnegie Mellon University.
   Google Scholar
• Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341–384. https://doi.org/10.3102/10769986027004341
   Web of Science ®Google Scholar
• Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. Paper presented at the 3rd International Workshop on Distributed Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/conferences/DSC-2003/Proceedings/Plummer.pdf
   Google Scholar
• Roberts, J. S., & Ma, Q. (2006). IRT models for the assessment of change across repeated measurements. In R. W. Lissitz (Ed.), Longitudinal and value added modeling of student performance (pp. 100–127). Maple Grove, MN: JAM Press.
   Google Scholar
• Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics, 12(4), 1151–1172. https://doi.org/10.1214/aos/1176346785
   Web of Science ®Google Scholar
• Sinharay, S. (2005). Assessing fit of unidimensional item response theory models using a Bayesian approach. Journal of Educational Measurement, 42(4), 375–394. https://doi.org/10.1111/j.1745-3984.2005.00021.x
   Web of Science ®Google Scholar
• Sinharay, S., Johnson, M. S., & Stern, H. S. (2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30(4), 298–321. https://doi.org/10.1177/0146621605285517
   Web of Science ®Google Scholar
• Stanford Center on Adolescence. (2014). Character development in adolescence. Retrieved from https://coa.stanford.edu/content/character-development-adolescence
   Google Scholar
• Su, Y. S., & Yajima, M. (2012). R2jags: A package for running jags from R. R Package Version 0.03-08.
   Google Scholar
• Verhelst, N. D., & Verstralen, H. H. (2001). An IRT model for multiple raters. In A. Boomsma, M. van Duijn, & T. Snijders (Eds.), Essays on item response theory (pp. 89–108). New York, NY: Springer-Verlag.
   Google Scholar
• Wilson, M., & Hoskens, M. (2001). The rater bundle model. Journal of Educational and Behavioral Statistics, 26(3), 283–306. https://doi.org/10.3102/10769986026003283
   Web of Science ®Google Scholar
• Wolfe, E. W. (2014). Methods for monitoring rating quality: Current practices and suggested changes. (White Paper). Iowa City, IA: Pearson Education.
   Google Scholar
• Xu, W., Huang, R., Zhang, H., El-Khamra, Y., & Walling, D. (2016). Empowering R with high performance computing resources for big data analytics. In R. Arora (Ed.), Conquering big data using high performance computing (pp. 191–217). New York, NY: Springer-Verlag.
   Google Scholar