Approaches to Item-Level Data with Cross-Classified Structure: An Illustration with Student Evaluation of Teaching

References

• Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573. https://doi.org/10.1007/BF02293814
   Web of Science ®Google Scholar
• Bacci, S., & Caviezel, V. (2011). Multilevel IRT models for the university teaching evaluation. Journal of Applied Statistics, 38(12), 2775–2791. https://doi.org/10.1080/02664763.2011.570316
   Web of Science ®Google Scholar
• Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 48. https://doi.org/10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801
   Web of Science ®Google Scholar
• Boring, A. (2017). Gender biases in student evaluations of teaching. Journal of Public Economics, 145, 27–41. https://doi.org/10.1016/j.jpubeco.2016.11.006
   Web of Science ®Google Scholar
• Brockx, B., Spooren, P., & Mortelmans, D. (2011). Taking the grading leniency story to the edge. The influence of student, teacher, and course characteristics on student evaluations of teaching in higher education. Educational Assessment, Evaluation and Accountability, 23(4), 289–306. https://doi.org/10.1007/s11092-011-9126-2
   Web of Science ®Google Scholar
• Cai, L. (2008). A Metropolis-Hastings Robbins-Monro algorithm for maximum likelihood nonlinear latent structure analysis with a comprehensive measurement model [Unpublished doctoral dissertation]. The University of North Carolina at Chapel Hill.
   Google Scholar
• Cai, L. (2010a). High-dimensional exploratory item factor analysis by a Metropolis–Hastings Robbins–Monro algorithm. Psychometrika, 75(1), 33–57. https://doi.org/10.1007/s11336-009-9136-x
   Web of Science ®Google Scholar
• Cai, L. (2010b). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335. https://doi.org/10.3102/1076998609353115
   Web of Science ®Google Scholar
• Chen, Y., & Hoshower, L. B. (2003). Student evaluation of teaching effectiveness: An assessment of student perception and motivation. Assessment & Evaluation in Higher Education, 28(1), 71–88. https://doi.org/10.1080/02602930301683
   Google Scholar
• Cho, S.-J., & Rabe-Hesketh, S. (2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics & Data Analysis, 55(1), 12–25. https://doi.org/10.1016/j.csda.2010.04.015
   Web of Science ®Google Scholar
• Chung, S., & Cai, L. (2021). Cross-classified random effects modeling for moderated item calibration. Journal of Educational and Behavioral Statistics, 46(6), 651–681. https://doi.org/10.3102/1076998620983908
   Web of Science ®Google Scholar
• Clayton, D., & Rasbash, J. (1999). Estimation in large cross random-effect models by data augmentation. Journal of the Royal Statistical Society Series A: Statistics in Society, 162(3), 425–436. https://doi.org/10.1111/1467-985X.00146
   Google Scholar
• Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1–22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x
   Web of Science ®Google Scholar
• Falk, C. F., & Cai, L. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21(3), 328–347. https://doi.org/10.1037/met0000059
   PubMed Web of Science ®Google Scholar
• Feistauer, D., & Richter, T. (2017). How reliable are students’ evaluations of teaching quality? A variance components approach. Assessment & Evaluation in Higher Education, 42(8), 1263–1279. https://doi.org/10.1080/02602938.2016.1261083
   Web of Science ®Google Scholar
• Feistauer, D., & Richter, T. (2018). Validity of students’ evaluations of teaching: Biasing effects of likability and prior subject interest. Studies in Educational Evaluation, 59, 168–178. https://doi.org/10.1016/j.stueduc.2018.07.009
   Web of Science ®Google Scholar
• Fisher, R. A. (1925). Theory of statistical estimation. Mathematical Proceedings of the Cambridge Philosophical Society, 22(5), 700–725. https://doi.org/10.1017/S0305004100009580
   Google Scholar
• Goldstein, H. (1986). Multilevel mixed linear model analysis using iterative generalized least squares. Biometrika, 73(1), 43–56. https://doi.org/10.1093/biomet/73.1.43
   Web of Science ®Google Scholar
• Houts, C. R., & Cai, L. (2020). flexmirt®: Flexible multilevel multidimensional item analysis and test scoring user’s manual version 3.5 [Computer software]. Vector Psychometric Group.
   Google Scholar
• Huang, S., & Cai, L. (2023). Cross-classified item response theory modeling with an application to student evaluation of teaching. Journal of Educational and Behavioral Statistics. 10769986231193351. https://doi.org/10.3102/10769986231193351
   Web of Science ®Google Scholar
• Kamata, A., & Vaughn, B. K. (2011). Multilevel item response theory modeling. In J. Hox & R. J. Kyle (Eds.), Handbook of advanced multilevel analysis (pp. 41–57). Routledge.
   Google Scholar
• Kember, D., & Leung, D. Y. (2008). Establishing the validity and reliability of course evaluation questionnaires. Assessment & Evaluation in Higher Education, 33(4), 341–353. https://doi.org/10.1080/02602930701563070
   Web of Science ®Google Scholar
• La Rocca, M., Parrella, M. L., Primerano, I., Sulis, I., & Vitale, M. P. (2017). An integrated strategy for the analysis of student evaluation of teaching: From descriptive measures to explanatory models. Quality & Quantity, 51(2), 675–691. https://doi.org/10.1007/s11135-016-0432-0
   Google Scholar
• Laube, H., Massoni, K., Sprague, J., & Ferber, A. L. (2007). The impact of gender on the evaluation of teaching: What we know and what we can do. NWSA Journal, 19(3), 87–104. http://www.jstor.org/stable/40071230
   Google Scholar
• Lemos, M., Queirós, C., Teixeira, P., & Menezes, I. (2011). Development and validation of a theoretically based, multidimensional questionnaire of student evaluation of university teaching. Assessment & Evaluation in Higher Education, 36(7), 843–864. https://doi.org/10.1080/02602938.2010.493969
   Web of Science ®Google Scholar
• Luo, W., & Kwok, O.-m. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behavioral Research, 44(2), 182–212. https://doi.org/10.1080/00273170902794214
   PubMed Web of Science ®Google Scholar
• Marsh, H. W. (1982). Seeq: A reliable, valid, and useful instrument for collecting students’ evaluations of university teaching. British Journal of Educational Psychology, 52(1), 77–95. https://doi.org/10.1111/j.2044-8279.1982.tb02505.x
   Web of Science ®Google Scholar
• Marsh, H. W. (1984). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential baises, and utility. Journal of Educational Psychology, 76(5), 707–754. https://doi.org/10.1037/0022-0663.76.5.707
   Web of Science ®Google Scholar
• Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. https://doi.org/10.1007/BF02296272
   Web of Science ®Google Scholar
• McCulloch, C. E., Searle, S. R., & Neuhaus, J. M. (2008). Generalized, linear, and mixed models. John Wiley & Sons.
   Google Scholar
• Meyers, J. L., & Beretvas, S. N. (2006). The impact of inappropriate modeling of cross-classified data structures. Multivariate Behavioral Research, 41(4), 473–497. https://doi.org/10.1207/s15327906mbr4104_3
   PubMed Web of Science ®Google Scholar
• Monroe, S., & Cai, L. (2014). Estimation of a Ramsay-curve item response theory model by the Metropolis–Hastings Robbins–Monro algorithm. Educational and Psychological Measurement, 74(2), 343–369. https://doi.org/10.1177/0013164413499344
   Web of Science ®Google Scholar
• Oon, P.-T., Spencer, B., & Kam, C. C. S. (2017). Psychometric quality of a student evaluation of teaching survey in higher education. Assessment & Evaluation in Higher Education, 42(5), 788–800. https://doi.org/10.1080/02602938.2016.1193119
   Web of Science ®Google Scholar
• Pagani, L., & Zanarotti, M. C. (2010). Some uses of Rasch models parameters in customer satisfaction data analysis. Quality Technology & Quantitative Management, 7(1), 83–95. https://doi.org/10.1080/16843703.2010.11673220
   Web of Science ®Google Scholar
• Patz, R. J., & Junker, B. W. (1999a). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24(4), 342–366. https://doi.org/10.2307/1165367
   Web of Science ®Google Scholar
• Patz, R. J., & Junker, B. W. (1999b). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178. https://doi.org/10.2307/1165199
   Web of Science ®Google Scholar
• R Core Team. (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
   Google Scholar
• Rampichini, C., Grilli, L., & Petrucci, A. (2004). Analysis of university course evaluations: From descriptive measures to multilevel models. Statistical Methods and Applications, 13(3), 357–373. https://doi.org/10.1007/s10260-004-0087-1
   Google Scholar
• Rantanen, P. (2013). The number of feedbacks needed for reliable evaluation. a multilevel analysis of the reliability, stability and generalisability of students’ evaluation of teaching. Assessment & Evaluation in Higher Education, 38(2), 224–239. https://doi.org/10.1080/02602938.2011.625471
   Web of Science ®Google Scholar
• Rasbash, J., & Goldstein, H. (1994). Efficient analysis of mixed hierarchical and cross-classified random structures using a multilevel model. Journal of Educational and Behavioral Statistics, 19(4), 337–350. https://doi.org/10.3102/10769986019004337
   Google Scholar
• Raudenbush, S. W. (1993). A crossed random effects model for unbalanced data with applications in cross-sectional and longitudinal research. Journal of Educational Statistics, 18(4), 321–349. https://doi.org/10.3102/10769986018004321
   Google Scholar
• Robbins, H., & Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586
   Google Scholar
• Royal, K. D., & Stockdale, M. R. (2015). Are teacher course evaluations biased against faculty that teach quantitative methods courses? International Journal of Higher Education, 4(1), 217–224. https://doi.org/10.5430/ijhe.v4n1p217
   Google Scholar
• Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1), 1–97. https://doi.org/10.1007/BF03372160
   Google Scholar
• Skrondal, A., & Rabe-Hesketh, S. (2014). Generalized latent variable modeling. Chapman & Hall/CRC.
   Google Scholar
• Spooren, P. (2010). On the credibility of the judge: A cross-classified multilevel analysis on students’ evaluation of teaching. Studies in Educational Evaluation, 36(4), 121–131. https://doi.org/10.1016/j.stueduc.2011.02.001
   Google Scholar
• Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598–642. https://doi.org/10.3102/0034654313496870
   Web of Science ®Google Scholar
• Sulis, I., & Capursi, V. (2013). Building up adjusted indicators of students’ evaluation of university courses using generalized item response models. Journal of Applied Statistics, 40(1), 88–102. https://doi.org/10.1080/02664763.2012.734796
   Web of Science ®Google Scholar
• Sulis, I., & Toland, M. D. (2017). Introduction to multilevel item response theory analysis: Descriptive and explanatory models. The Journal of Early Adolescence, 37(1), 85–128. https://doi.org/10.1177/0272431616642328
   Web of Science ®Google Scholar
• Tran, T. T. T., & Do, T. X. (2022). Student evaluation of teaching: Do teacher age, seniority, gender, and qualification matter? Educational Studies, 48(4), 443–470. https://doi.org/10.1080/03055698.2020.1771545
   Web of Science ®Google Scholar
• Yang, J. S., & Cai, L. (2014). Estimation of contextual effects through nonlinear multilevel latent variable modeling with a Metropolis–Hastings Robbins–Monro algorithm. Journal of Educational and Behavioral Statistics, 39(6), 550–582. https://doi.org/10.3102/1076998614559972
   Web of Science ®Google Scholar
• Zabaleta, F. (2007). The use and misuse of student evaluations of teaching. Teaching in Higher Education, 12(1), 55–76. https://doi.org/10.1080/13562510601102131
   Web of Science ®Google Scholar
• Zhao, J., & Gallant, D. J. (2012). Student evaluation of instruction in higher education: Exploring issues of validity and reliability. Assessment & Evaluation in Higher Education, 37(2), 227–235. https://doi.org/10.1080/02602938.2010.523819
   Web of Science ®Google Scholar