Using Topic Models to Understand Rater-Mediated Writing Assessments

References

• Behizadeh, N., & Pang, E.P. (2016). Awaiting a new wave: The status of state writing assessment in the United States. Assessing Writing, 29, 25–41. https://doi.org/10.1016/j.asw.2016.05.003
   Web of Science ®Google Scholar
• Blei, D.M., & Lafferty, J.D. (2009). Topic models. In A.N. Shrivastava & M. Sahami (Eds.), Text mining (pp. 101–124). Chapman and Hall/CRC.
   Google Scholar
• Blei, D.M., Ng, A.Y., & Jordan, M.I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan), 993–1022.
   Google Scholar
• Boyd-Graber, J., Mimno, D.., & D., Newman. (2014). Care and feeding of topic models: Problems, diagnostics, and improvements. In E.M. Airoldi, D. Blei, E.A. Erosheva, & S.E. Fienberg (Eds.), Handbook of mixed membership models and its applications (pp. 225–254). CRC Press.
   Google Scholar
• Cardozo-Gaibisso, L., Kim, S., Buxton, C., & Cohen, A. (2020). Thinking beyond the score: Multidimensional analysis of student performance to inform the next generation of science assessments. Journal of Research in Science Teaching, 57(6), 856–878. https://doi.org/10.1002/tea.21611
   Web of Science ®Google Scholar
• Choi, H.J., Kwak, M., Kim, S., Xiong, J., Cohen, A.S., & Bottge, B.A. (2017). An application of a topic model to two educational assessments. In M. Wiberg, S. Culpepper, R. Janssen, J. González, & D. Molenaar (Eds.), Quantitative psychology: proceedings of the psychometric society (pp. 449–459). Springer.
   Google Scholar
• Copur-Gencturk, Y., Choi, H.J., & Cohen, A. (2022). Investigating teachers’ understanding through topic modeling: a promising approach to studying teachers’ knowledge. Journal of Mathematics Teacher Education, 1–22.
   Web of Science ®Google Scholar
• Deveaud, R., SanJuan, E., & Bellot, P. (2014). Accurate and effective latent concept modeling for ad hoc information retrieval. Document Numérique, 17(1), 61–84. https://doi.org/10.3166/dn.17.1.61-84
   Google Scholar
• Duong, E., Mellom, P., & Hixon, R. (2019). Using topic modeling to analyze the effects of instructional conversation on 3rd grade students’ writing [Paper presentation]. Annual Meeting of the American Association for Applied Linguistics, Atlanta, GA. March 9, 2019.
   Google Scholar
• Georgia Center for Assessment. (2021). Administration resource guide. Unpublished document.
   Google Scholar
• Grün, B., & Hornik, K. (2011). topicmodels: An R package for fitting topic models. Journal of Statistical Software, 40(13), 1–30. https://doi.org/10.18637/jss.v040.i13
   Web of Science ®Google Scholar
• Kim, S., Kwak, M., Cardozo-Gaibisso, L.A., Buxton, C.A., & Cohen, A.S. (2017). Statistical and qualitative analyses of students’ answers to a constructed response test of science inquiry knowledge. Journal of Writing Analytics, 1(1), 82–102. https://doi.org/10.37514/JWA-J.2017.1.1.05
   Google Scholar
• Kim, S.-H., & Cohen, A.S. (2007, July). A brief history of testing in the ancient world and its implications today [Paper presentation]. International Meeting of the Psychometric Society, Tokyo, Japan.
   Google Scholar
• Lane, S., & Stone, C. (2006). Performance assessment. In R.L. Brennan (Ed.), Educational measurement (4th ed., pp. 388–431) American Council on Education and Praeger.
   Google Scholar
• Mardones-Segovia, C., Choi, H.J., Hong, M., Wheeler, J.M., Cohen, A.S. (2022). Comparison of Estimation Algorithms for Latent Dirichlet Allocation. In: M. Wiberg, D. Molenaar, J. González, J.S. Kim, H. Hwang, (Eds.), Quantitative Psychology. IMPS 2021. Springer Proceedings in Mathematics & Statistics, vol 393. Springer, Cham. https://doi.org/10.1007/978-3-031-04572-1_3
   Google Scholar
• Porteous, I., Newman, D., Ihler, A., Asuncion, A., Smyth, P., & Welling, M. (2008, August). Fast collapsed Gibbs sampling for latent Dirichlet allocation [Paper presentation]. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 569–577). https://doi.org/10.1145/1401890.1401960
   Google Scholar
• Schofield, A., Magnusson, M., Mimno, D. (2017). Understanding text pre-processing for latent Dirichlet allocation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain (Vol. 2, pp. 432–436).
   Google Scholar
• Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464.
   Web of Science ®Google Scholar
• Shermis, M.D., Mao, L., Mulholland, M., & Kieftenbeld, V. (2017). Use of automated scoring features to generate hypotheses regarding language-based DIF. International Journal of Testing, 17(4), 351–371. https://doi.org/10.1080/15305058.2017.1308949
   Web of Science ®Google Scholar
• Shin, J., Guo, Q., & Gierl, M.J. (2019). Multiple-choice item distractor development using topic modeling approaches. Frontiers in Psychology, 10, 825.
   PubMed Web of Science ®Google Scholar
• Spiegelhalter, D.J., Best, N.G., Carlin, B.P., & Van Der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583–639. https://doi.org/10.1111/1467-9868.00353
   Web of Science ®Google Scholar
• Weigle, S.C., & Montee, M. (2012). Raters’ perceptions of textual borrowing in integrated writing tasks. Studies in Writing, 27, 117–152.
   Google Scholar
• Weigle, S.C., Yang, W., & Montee, M. (2013). Exploring reading processes in an academic reading test using short-answer questions. Language Assessment Quarterly, 10(1), 28–48. https://doi.org/10.1080/15434303.2012.750660
   Web of Science ®Google Scholar
• Wheeler, J.M., Engelhard, G., & Wang, J. (2022). Exploring rater accuracy using unfolding models combined with topic models: Incorporating supervised latent Dirichlet allocation. Measurement: Interdisciplinary Research and Perspectives, 20(1), 34–46. https://doi.org/10.1080/15366367.2021.1915094
   Web of Science ®Google Scholar
• Wheeler, J.M., Wang, S., Tan, Y., & Cohen, A.S. (2022). Textual Data as Process Data: A New Scoring Procedure for Mixed-Format Assessments. Paper presented at the Annual Meeting of the National Council on Measurement in Education. San Diego, Ca.
   Google Scholar
• Wilbur, W.J., & Sirotkin, K. (1992). The automatic identification of stop words. Journal of Information Science, 18(1), 45–55. https://doi.org/10.1177/016555159201800106
   Web of Science ®Google Scholar
• Wind, S.A., Wolfe, E.W., Engelhard, G., Jr., Foltz, P., & Rosenstein, M. (2018). The influence of rater effects in training sets on the psychometric quality of automated scoring for writing assessments. International Journal of Testing, 18(1), 27–49. https://doi.org/10.1080/15305058.2017.1361426
   Web of Science ®Google Scholar
• Xiong, J., Choi, H.-J., Kim, S., Kwak, M., & Cohen, A.S. (2019). Topic modeling of constructed-response answers on social study assessments. In M. Wiberg, S. Culpepper, R. Janssen, J. González, & D. Molenaar (Eds.), Quantitative Psychology: Proceedings of the 84th Annual Meeting of the International Psychometric Society, Springer (pp. 263–274).
   Google Scholar
• Xiong, J., Wheeler, J.M., Choi, H.J., Lee, J., & Cohen, A.S. (2021). An empirical study of developing automated scoring engine using supervised latent dirichlet allocation. In M. Wiberg, U. Böckenholt, J. González, D. Molenaar, & J.-S. Kim (Eds.), Quantitative Psychology: Proceedings of the 85th Annual Meeting of the International Psychometric Society, Springer (pp. 429–438).
   Google Scholar
• Zhang, M., Williamson, D.M., Breyer, F.J., & Trapani, C. (2012). Comparison of e-rater® automated essay scoring model calibration methods based on distributional targets. International Journal of Testing, 12(4), 345–364. https://doi.org/10.1080/15305058.2011.645973
   Google Scholar