Automated summary evaluation with inbuilt rubric method: An alternative to constructed responses and multiple-choice tests assessments

References

• Abad, F. J., J. Olea, V. Ponsoda, and C. García. 2011. Medición en ciencias sociales y de la salud [Measurement in social and health sciences]. Madrid: Síntesis.
   Google Scholar
• Ashenafi, M. M. 2017. “Peer-assessment in Higher Education–twenty-first Century Practices, challenges and the Way Forward.” Assessment & Evaluation in Higher Education 42 (2):226–51. doi: 10.1080/02602938.2015.1100711.
   Web of Science ®Google Scholar
• Asimov, I. 1969. Great ideas of science. Boston: Houghton Mifflin.
   Google Scholar
• Baartman, L. K., T. J. Bastiaens, P. A. Kirschner, and C. P. van der Vleuten. 2007. “Evaluating Assessment Quality in Competence-based Education: A Qualitative Comparison of Two Frameworks.” Educational Research Review 2 (2):114–29. doi: 10.1016/j.edurev.2007.06.001.
   Google Scholar
• Bejar, I. I., R. J. Mislevy, and M. Zhang. 2016. Automated Scoring with Validity in Mind. In A.A. Rupp & J.P. Leighton (Eds.), The wiley handbook of cognition and assessment: Frameworks, Methodologies, and applications. (pp. 226–246). Oxford: Wiley Blackwell. https://dx.doi.org/10.1002/9781118956588.ch10.
   Google Scholar
• Dawson, P. 2017. “Assessment Rubrics: towards Clearer and More Replicable Design, research and Practice.” Assessment & Evaluation in Higher Education 42 (3):347–60. doi: 10.1080/02602938.2015.1111294.
   Web of Science ®Google Scholar
• Dawson, P., M. Henderson, P. Mahoney, M. Phillips, T. Ryan, D. Boud, and E. Molloy. 2019. “What Makes for Effective Feedback: staff and Student Perspectives.” Assessment & Evaluation in Higher Education Advance online publication 44(1):25-36. 10.1080/02602938.2018.1467877
   Web of Science ®Google Scholar
• Foltz, P. W., D. Laham, and T. K. Landauer. 1999. Automated Essay Scoring: Applications to Educational Technology. In B. Collis & R. Oliver (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia & Telecommunications (ED-MEDIA '99) (pp. 939–944). Seattle, USA: Association for the Advancement of Computing in Education (AACE).
   Google Scholar
• Franzke, M., E. Kintsch, D. Caccamise, N. Johnson, and S. Dooley. 2005. “Summary Street: Computer Support for Comprehension and Writing.” Journal of Educational Computing Research 33 (1):53–80. doi: 10.2190/DH8F-QJWM-J457-FQVB.
   Google Scholar
• Hong, W. 2016. “The Effect of Summarizing Task and Interaction on Korean Middle School Students’ Reading Comprehension.” Studies in English Education 21 (1):39–71. doi: 10.22275/SEE.21.1.02.
   Google Scholar
• Hu, X., Z. Cai, P. Wiemer-Hastings, A. C. Graesser, and D. S. McNamara. 2007. Strengths, limitations, and extensions of LSA. In T.K. Landauer, D.S., McNamara, S. Dennis, & W. Kintsch, Handbook of latent semantic analysis. (pp. 401–426). New Jersey: Routledge. https://dx.doi.org/10.4324/9780203936399.ch20.
   Google Scholar
• Jones, M. N., J. Willits, and S. Dennis. 2015. Models of semantic memory. In J.R. Busemeyer, Z. Wang, J.T. Townsend, & A. Eidels (Eds.), The oxford handbook of mathematical and computational psychology. (pp. 232–254). New York: Oxford University Press. https://dx.doi.org/10.1111/bjop.12201.
   Google Scholar
• Jonsson, A., and G. Svingby. 2007. “The Use of Scoring Rubrics: Reliability, validity and Educational Consequences.” Educational Research Review 2 (2):130–44. doi: 10.1016/j.edurev.2007.05.002.
   Google Scholar
• Jorge-Botana, G., R. Olmos, and A. Barroso. 2013. July). Gallito 2.0: A Natural Language Processing tool to support Research on Discourse. Proceedings of the Twenty-third Annual Meeting of the Society for Text and Discourse, Valencia.
   Google Scholar
• Kintsch, E., D. Steinhart, G. Stahl, L. S. A. R. G. Lsa Research Group, C. Matthews, and R. Lamb. 2000. “Developing Summarization Skills through the Use of LSA-based Feedback.” Interactive Learning Environments 8 (2):87–109. doi: 10.1076/1049-4820(200008)8:2;1-B;FT087.
   Google Scholar
• Klein, R., A. Kyrilov, and M. Tokman. 2011. June). Automated Assessment of Short Free-Text Responses in Computer Science using Latent Semantic Analysis. Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (ITiCSE '11). https://dx.doi.org/10.1145/1999747.1999793.
   Google Scholar
• Landauer, T. K., P. W. Foltz, and D. Laham. 1998. “An Introduction to Latent Semantic Analysis.” Discourse Processes 25 (2-3):259–84. doi: 10.1080/01638539809545028.
   Web of Science ®Google Scholar
• Landauer, T. K., D. S. McNamara, S. Dennis, and W. Kintsch. 2007. The handbook of latent semantic analysis. New Jersey: Routledge. doi/10.4324/9780203936399.
   Google Scholar
• Landauer, T. K., and S. Dumais. 1997. “A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of the Acquisition, induction, and Representation of Knowledge.” Psychological Review 104 (2):211–40. doi: 10.1037/0033-295X.104.2.211.
   Web of Science ®Google Scholar
• León, J. A., R. Olmos, I. Escudero, J. J. Cañas, and L. Salmerón. 2006. “Assessing Short Summaries with Human Judgments Procedure and Latent Semantic Analysis in Narrative and Expository Texts.” Behavior Research Methods 38 (4):616–27. doi: 10.3758/BF03193894.
   PubMed Web of Science ®Google Scholar
• Martín-Loeches, M. 2016. Origen y Evolución del Lenguaje Humano: Una Perspectiva Neurocognitiva [Origin and Evolution of Human Language: A Neurocognitive Perspective]. Retrieved from http://www.atapuerca.org/ficha/ZE7D1307E-A298-9B9E-5CF101F70223C275/origen-y-evolucion-del-lenguaje-humano-una-perspectiva-neurocognitive.
   Google Scholar
• Martínez-Huertas, J. A., O. Jastrzebska, A. Mencu, J. Moraleda, R. Olmos, and J. A. León. 2018. “Analyzing Two Automatic Assessment LSA´s Methods (Golden Summary vs Inbuilt Rubric) in Summaries Extracted from Expository Texts.” Psicología Educativa 24 (2):85–92. http://doi.org/10.5093/psed2048a9. doi: 10.5093/psed2048a9.
   Web of Science ®Google Scholar
• McGrath, R. E., and G. J. Meyer. 2006. “When Effect Sizes Disagree: the Case of r and d.” Psychological Methods 11 (4):386. doi: 10.1037/1082-989X.11.4.386.
   PubMed Web of Science ®Google Scholar
• McNamara, D. S. 2011. “Computational Methods to Extract Meaning from Text and Advance Theories of Human Cognition.” Topics in Cognitive Science 3 (1):3–17. doi: 10.1111/j.1756-8765.2010.01117.x.
   PubMed Web of Science ®Google Scholar
• Millis, K., J. Magliano, K. Wiemer-Hastings, S. Todaro, and D. S. McNamara. 2007. Assessing and improving comprehension with latent semantic analysis. In T.K. Landauer, D.S., McNamara, S. Dennis, & W. Kintsch, Handbook of latent semantic analysis. (pp. 207–225). New Jersey: Routledge. https://dx.doi.org/10.4324/9780203936399.ch11.
   Google Scholar
• Nakov, P., A. Popova, and P. Mateev. 2001. September). Weight Functions Impact on LSA Performance. Paper presented at the EuroConference Recent Advances in Natural Language Processing (RANLP’01). Sophia, Bulgary.
   Google Scholar
• Olmos, R., G. Jorge-Botana, J. A. León, and I. Escudero. 2014. “Transforming Selected Concepts into Dimensions in Latent Semantic Analysis.” Discourse Processes 51 (5-6):494–510. doi: 10.1080/0163853X.2014.913416.
   Web of Science ®Google Scholar
• Olmos, R., G. Jorge-Botana, J. M. Luzón, J. Cordero, and J. A. León. 2016. “Transforming LSA Space Dimensions into a Rubric for an Automatic Assessment and Feedback System.” Information Processing & Management 52 (3):359–73. doi: 10.1016/j.ipm.2015.12.002.
   Web of Science ®Google Scholar
• Panadero, E., and A. Jonsson. 2013. “The Use of Scoring Rubrics for Formative Assessment Purposes Revisited: A Review.” Educational Research Review 9:129–44. doi: 10.1016/j.edurev.2013.01.002.
   Web of Science ®Google Scholar
• Raykov, T., and G. A. Marcoulides. 2008. An introduction to applied multivariate analysis. Routledge: New York. https://dx.doi.org/10.1111/j.1751-5823.2009.00074_18.x.
   Google Scholar
• Reddy, Y. M., and H. Andrade. 2010. “A Review of Rubric Use in Higher Education.” Assessment & Evaluation in Higher Education 35 (4):435–48. doi: 10.1080/02602930902862859.
   Web of Science ®Google Scholar
• Saddler, B., K. Asaro-Saddler, M. Moeyaert, and T. Ellis-Robinson. 2017. “Effects of a Summarizing Strategy on Written Summaries of Children with Emotional and Behavioral Disorders.” Remedial and Special Education 38 (2):87–97. doi: 10.1177/0741932516669051.
   Web of Science ®Google Scholar
• Seifert, T., and O. Feliks. 2019. “Online Self-assessment and Peer-assessment as a Tool to Enhance Student-teachers’ Assessment Skills.” Assessment & Evaluation in Higher Education 44 (2):169–185. doi: 10.1080/02602938.2018.1487023.
   Web of Science ®Google Scholar
• Shapiro, A. M., and D. S. McNamara. 2000. “The Use of Latent Semantic Analysis as a Tool for the Quantitative Assessment of Understanding and Knowledge.” Journal of Educational Computing Research 22 (1):1–36. doi: 10.2190/M811-G475-WKMX-X0JH.
   Web of Science ®Google Scholar
• Shermis, M. D., J. Burstein, D. Higgins, and K. Zechner. 2010. Automated essay scoring: Writing assessment and instruction. In E. Baker, B. McGaw, & N.S. Petersen (Eds.), International encyclopedia of education., Vol.4 (3rd ed.) (pp. 20–26). Oxford: Elsevier. https://dx.doi.org/10.1016/B978-0-08-044894-7.00233-5.
   Google Scholar
• Shrout, P., and J. Fleiss. 1979. “Intraclass Correlations: Uses in Assessing Rater Reliability.” Psychological Bulletin 86 (2):420–8. doi: 10.1037/0033-2909.86.2.420.
   PubMed Web of Science ®Google Scholar
• Stevens, E. A., S. Park, and S. Vaughn. 2018. “A Review of Summarizing and Main Idea Interventions for Struggling Readers in Grades 3 through 12: 1978–2016.” Remedial and Special Education. Advance Online Publication. doi: 10.1177/0741932517749940.
   Web of Science ®Google Scholar
• Sung, Y. T., C. N. Liao, T. H. Chang, C. L. Chen, and K. E. Chang. 2016. “The Effect of Online Summary Assessment and Feedback System on the Summary Writing on 6th Graders: The LSA-based Technique.” Computers & Education 95:1–18. doi: 10.1016/j.compedu.2015.12.003.
   Web of Science ®Google Scholar
• van Dijk, T. A., and W. Kintsch. 1983. Strategies of discourse comprehension. New York: Academic Press.
   Google Scholar
• Wade-Stein, D., and E. Kintsch. 2004. “Summary Street: Interactive Computer Support for Writing.” Cognition and Instruction 22 (3):333–62. doi: 10.1207/s1532690xci2203_3.
   Web of Science ®Google Scholar