Organisational quality of ESL argumentative essays and its influence on pre-service teachers’ judgments

References

• Alderson, J. C., Brunfaut, T., & Harding, L. (2014). Towards a theory of diagnosis in second and foreign language assessment: Insights from professional practice across diverse fields. Applied Linguistics, 36(2), 236–21. https://doi.org/10.1093/applin/amt046
   Web of Science ®Google Scholar
• Bacha, N. (2001). Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29(3), 371–383. https://doi.org/10.1016/S0346-251X(01)00025-2
   Google Scholar
• Bates, C., & Nettelbeck, T. (2001). Primary school teachers‘ judgements of reading achievement. Educational Psychology, 21(2), 177–187. https://doi.org/10.1080/01443410020043878
   Google Scholar
• Baumert, J., & Kunter, M. (2006). Keyword: Professional competencies of teachers. Zeitschrift Für Erziehungswissenschaften, 4(9), 469–520. https://doi.org/10.1007/s11618-006-0165-2
   Google Scholar
• Bechger, T. M., Maris, G., & Hsiao, Y. P. (2010). Detecting halo effects in performance-based examinations. Applied Psychological Measurement, 34(8), 607–619. https://doi.org/10.1177/0146621610367897
   Web of Science ®Google Scholar
• Birkel, P., & Birkel, C. (2002). How concordant are teachers’ essay scorings? A replication of Rudolf Weiss’ studies. Psychologie in Erziehung Und Unterricht, 49(3), 219–224.
   Web of Science ®Google Scholar
• Blömeke, S., Gustafsson, J.-E., & Shavelson, R. J. (2015). Beyond dichotomies: Competence viewed as a continuum. Zeitschrift Für Psychologie, 223(1), 3–13. https://doi.org/10.1027/2151-2604/a000194
   Web of Science ®Google Scholar
• Bridge, C. A., & Winograd, P. N. (1982). Readers’ awareness of cohesive relationships during cloze comprehension. Journal of Reading Behavior, XIV(3), 299–312. https://doi.org/10.1080/10862968209547457
   Google Scholar
• Charney, D. (1984). The validity of using holistic scoring to evaluate writing: A critical overview. Research in the Teaching of English, 18(1), 65–81.
   Web of Science ®Google Scholar
• Cheng, X., & Steffensen, M. S. (1996). Metadiscourse: A technique for improving student writing. Research in the Teaching of English, 30(2), 149–181.
   Web of Science ®Google Scholar
• Chiang, S. (2003). The importance of cohesive conditions to perceptions of writing quality at the early stages of foreign language learning. System, 31(4), 471–484. https://doi.org/10.1016/j.system.2003.02.002
   Google Scholar
• Connor, U. (1990). Linguistic/rhetorical measures for international persuasive student writing. Research in the Teaching of English, 24(1), 67–87.
   Web of Science ®Google Scholar
• Council of Europe. (2001) . Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press.
   Google Scholar
• Crismore, A., Markkanen, R., & Steffensen, M. S. (1993). Metadiscourse in persuasive writing. Written Communication, 10(1), 39–71. https://doi.org/10.1177/0741088393010001002
   Web of Science ®Google Scholar
• Crossley, S. A., Allen, L. K., Kyle, K., & McNamara, D. S. (2014). Analyzing discourse processing using a simple natural language processing tool. Discourse Processes, 51(5–6), 511–534. https://doi.org/10.1080/0163853X.2014.910723
   Web of Science ®Google Scholar
• Crossley, S. A., Kyle, K., & McNamara, D. S. (2015). To aggregate or not? Linguistic features in automatic essay scoring and feedback systems. The Journal of Writing Assessment, 8(1), 1-19.
   Google Scholar
• Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 32(1), 1–16. https://doi.org/10.1016/j.jslw.2016.01.003
   Web of Science ®Google Scholar
• Crossley, S. A., & McNamara, D. S. (2009). Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18(2), 119–135. https://doi.org/10.1016/j.jslw.2009.02.002
   Web of Science ®Google Scholar
• Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35(2), 115–135. https://doi.org/10.1111/j.1467-9817.2010.01449.x
   Web of Science ®Google Scholar
• Crusan, D. (2010). Assessment in the second language writing classroom. University of Michigan.
   Google Scholar
• Culham, R. (2003). 6 + 1 traits of writing: The complete guide. Scholastic.
   Google Scholar
• Dempsey, M. S., PytlikZillig, L. M., & Bruning, R. H. (2009). Helping preservice teachers learn to assess writing: Practice and feedback in a Web-based environment. Assessing Writing, 14(1), 38–61. https://doi.org/10.1016/j.asw.2008.12.003
   Google Scholar
• Edelenbos, P., & Kubanek-German, A. (2004). Teacher assessment: The concept of ‘diagnostic competence’. Language Testing, 21(3), 259–283. https://doi.org/10.1191/0265532204lt284oa
   Google Scholar
• Erdosy, U. M. (2004). Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions (Research Report (RR-03-17). Educational Testing Service.
   Google Scholar
• Feinberg, A. B., & Shapiro, E. S. (2003). Accuracy of teacher judgments in predicting oral reading fluency. School Psychology Quarterly, 18(1), 52–65. https://doi.org/10.1521/scpq.18.1.52.20876
   Web of Science ®Google Scholar
• Freedman, S. W. (1979). How characteristics of student essays influence teachers‘ evaluations. Journal of Educational Psychology, 71(3), 328–338. https://doi.org/10.1037/0022-0663.71.3.328
   Web of Science ®Google Scholar
• Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193–202. https://doi.org/10.3758/BF03195564
   PubMedGoogle Scholar
• Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238. https://doi.org/10.1016/j.asw.2013.05.002
   Web of Science ®Google Scholar
• Halliday, M. A. K., & Hasan, R. (2013). Cohesion in English. Routledge.
   Google Scholar
• Hyland, K. (1990). A Genre Description of the Argumentative Essay. RELC Journal, 21(1), 66–78. https://doi.org/10.1177/003368829002100105
   Google Scholar
• Hyland, K. (1998). Persuasion and context: The pragmatics of academic metadiscourse. Journal of Pragmatics, 30(4), 437–455. https://doi.org/10.1016/S0378-2166(98)00009-5
   Web of Science ®Google Scholar
• Hyland, K. (2005). Metadiscourse. Continuum.
   Google Scholar
• Hyland, K. (2008). Second Language Writing. Cambridge University Press.
   Google Scholar
• Hyland, K., & Tse, P. (2004). Metadiscourse in academic writing: A reappraisal. Applied Linguistics, 25(2), 156–177. https://doi.org/10.1093/applin/25.2.156
   Web of Science ®Google Scholar
• Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B. (1981). Testing ESL composition: A practical approach. Newbury House Publishers.
   Google Scholar
• Jalilifar, A., & Alipour, M. (2007). How explicit instruction makes a difference: Metadiscourse markers and EFL learners‘ reading comprehension skill. Journal of College Reading and Learning, 38(1), 35–52. https://doi.org/10.1080/10790195.2007.10850203
   Google Scholar
• Jansen, T., Vögelin, C., Machts, N., Keller, S. D., & Köller, O. (submitted). Matching the machine: Comparing teachers’ and student teachers’ judgments on student essays to standardised ratings.
   Google Scholar
• Kaiser, J., Südkamp, A., & Möller, J. (2016). The effects of student characteristics on teachers’ judgment accuracy: Disentangling ethnicity, minority status, and achievement. Journal of Educational Psychology, 109(6), 871–888. https://doi.org/10.1037/edu0000156
   Web of Science ®Google Scholar
• Keller, S. D. (2013). Integrative Schreibdidaktik Englisch für die Sekundarstufe—Theorie, Prozessgestaltung, Empirie. Narr Verlag.
   Google Scholar
• Knoch, U. (2007). ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing, 12(2), 108–128. https://doi.org/10.1016/j.asw.2007.07.002
   Google Scholar
• Knoch, U. (2009). Diagnostic writing assessment—The development and validation of a rating scale. Lang.
   Google Scholar
• Knopp, M., Jost, J., Nachtwei, N., Becker-Mrotzek, M., & Grabowski, J. (2012). Teilkomponenten von Schreibkompetenz untersuchen: Bericht aus einem interdisziplinären empirischen Projekt. In H. Bayrhuber, U. Harms, B. Muszynski, B. Ralle, M. Rothgangel, L.-H. Schön, H. J. Vollmer, & H.-G. Weigand (Eds.), Formate Fachdidaktischer Forschung—Empirische Projekte—Historische Analysen—Theoretische Grundlegungen (Vols. 47–65). Waxmann.
   Google Scholar
• Kruse, O. (2013). Perspectives on academic writing in european higher education: Genres, practices, and competences. Revista de Docencia Universitaria, 11(1), 37–58. https://doi.org/10.4995/redu.2013.5591
   Google Scholar
• Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: indices, tools, findings, and application. TESOL Quarterly, 49(4), 757–786. https://doi.org/10.1002/tesq.194
   Web of Science ®Google Scholar
• Lee, I. (2002). Teaching coherence to ESL students: A classroom inquiry. Journal of Second Language Writing, 11(2), 135–159. https://doi.org/10.1016/S1060-3743(02)00065–6
   Web of Science ®Google Scholar
• Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276. https://doi.org/10.1191/0265532202lt230oa
   Google Scholar
• Marshall, J. C., & Powers, J. M. (1969). Writing neatness, composition errors, and essay grades. Journal of Educational Measurement, 6(2), 97–101. https://doi.org/10.1111/j.1745-3984.1969.tb00665.x
   Web of Science ®Google Scholar
• McColloy, W., & Remsted, R. (1965). Composition rating scale for general merit: An experimental evaluation. Journal of Educational Research, LIX (2), 55–57. https://doi.org/10.1080/00220671.1965.10883300
   Google Scholar
• McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27(1), 57–86. https://doi.org/10.1177/0741088309351547
   Web of Science ®Google Scholar
• McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
   Google Scholar
• McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–288. https://doi.org/10.1080/01638539609544975
   Web of Science ®Google Scholar
• Messick, S. (1994). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Educational Testing Service.
   Google Scholar
• National Assessment Governing Board. (2010). Writing framework for the 2011 national assessment of education progress. US Government Printing Office.
   Google Scholar
• National Statistical Office of Switzerland. (2016). Graduation rates – data of figures. Retrieved from https://www.bfs.admin.ch/bfs/de/home/statistiken/bildung-wissenschaft/bildungsindikatoren/bildungssystem-schweiz/themen/abschluesse/maturitaetsquote.assetdetail.4282892.html
   Google Scholar
• Nimehchisalem, V. (2010). Materials for assessing the writing skill. Advances in Language and Literary Studies, 1(2), 233–255. https://doi.org/10.7575/aiac.alls.v.1n.1p.233
   Google Scholar
• Parr, J. M., & Timperley, H. S. (2010). Feedback to writing, assessment for teaching and learning and student progress. Assessing Writing, 15(2), 68–85. https://doi.org/10.1016/j.asw.2010.05.004
   Google Scholar
• Porsch, R. (2010). Schreibkompetenzvermittlung im Englischunterricht in der Sekundarstufe I - Empirische Analysen zu Leistungen, Einstellungen, Unterrichtsmethoden und Zusammenhängen von Leistungen in der Mutter- und Fremdsprache. Waxmann.
   Google Scholar
• Rakedzon, T., & Baram-Tsabari, A. (2017). To make a long story short: A rubric for assessing graduate students’ academic and popular science writing skills. Assessing Writing, 32, 28–42. https://doi.org/10.1016/j.asw.2016.12.004
   Web of Science ®Google Scholar
• Rauin, U., & Meier, U. (2007). Subjektive Einschätzungen des Kompetenzerwerbs in der Lehramtsausbildung. In M. Lüders & J. Wissinger (Eds.), Forschung zur Lehrerbildung—Kompetenzentwicklung und Programmevaluation (pp. 103–131). Waxmann.
   Google Scholar
• Ready, D. D., & Wright, D. L. (2011). Accuracy and inaccuracy in teachers’ perceptions of young children’s cognitive abilities: The role of child background and classroom context. American Educational Research Journal, 48(2), 335–360. https://doi.org/10.3102/0002831210374874
   Web of Science ®Google Scholar
• Rezaei, A. R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15(1), 18–39. https://doi.org/10.1016/j.asw.2010.01.003
   Google Scholar
• Ruegg, R., & Sugiyama, Y. (2013). Organization of ideas in writing: What are raters sensitive to? Language Testing in Asia, 3(1), 1–13. https://doi.org/10.1186/2229-0443-3-8
   Google Scholar
• Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119–144. https://doi.org/10.1007/BF00117714
   Web of Science ®Google Scholar
• Scannell, D. P., & Marshall, J. C. (1966). The effect of selected composition errors on grades assigned to essay examinations. American Educational Research Journal, 3(2), 125–130. https://doi.org/10.3102/00028312003002125
   Web of Science ®Google Scholar
• Schmider, E., Ziegler, M., Danay, E., Beyer, L., & Bühner, M. (2010). Is it really robust? Reinvestigating the robustness of ANOVA against violations of the normal distribution assumption. Methodology, 6(4), 147–151. https://doi.org/10.1027/1614-2241/a000016
   Web of Science ®Google Scholar
• Schrader, F.-W. (2013). Teacher diagnosis and diagnostic competence. Beiträge Zur Lehrerbildung, 31(2), 154–165.
   Google Scholar
• Südkamp, A., & Praetorius, A.-K. (2017). Diagnostische Kompetenz von Lehrkräften: Theoretische und methodische Weiterentwicklungen. Waxmann.
   Google Scholar
• Swales, J. (1990). Genre analysis: English in academic and research settings. Cambridge University Press.
   Google Scholar
• Tejada, Á. Z., Gallardo, C. N., Ferradá, C. M., & Calderón, L. I. (2015). 2L english texts and cohesion in upper CEFR levels: A corpus-based approach. Procedia Social and Behavioral Sciences, 212, 192–197. https://doi.org/10.1016/j.sbspro.2015.11.319
   Google Scholar
• Thorndike, E. L. (1920). A constant error in psychological ratings. Journal of Applied Psychology, 33, 263–271. https://doi.org/10.1037/h0071663
   Google Scholar
• Vande Kopple, W. J. (1985). Some exploratory discourse on metadiscourse. College Composition and Communication, 36(1), 82–93. https://doi.org/10.2307/357609
   Web of Science ®Google Scholar
• Vande Kopple, W. J. (1997, March 12). Refining and applying views of metadiscourse [Paper presentation]. Annual Meeting of the Conference on College Composition and Communication, Phoenix, AZ, USA.
   Google Scholar
• Vögelin, C., Jansen, T., Keller, S. D., Machts, N., & Möller, J. (2019). The influence of lexical features on teacher judgements of ESL argumentative essays. Assessing Writing, 39, 50–63. https://doi.org/10.1016/j.asw.2018.12.003
   Web of Science ®Google Scholar
• Vögelin, C., Jansen, T., Keller, S. D., & Möller, J. (2018). The impact of vocabulary and spelling on judgments of ESL essays: An analysis of teacher comments. The Language Learning Journal, 1–17. https://doi.org/10.1080/09571736.2018.1522662
   Google Scholar
• Weigle, S. C. (2002). Assessing writing. Cambridge University Press.
   Google Scholar
• Weir, C. (1988). The specification, realization and validation of an english language proficiency test. In A. Hughes & T. B. Council (Ed.), Testing English for University Study. ELT Documents 127 (pp. 45–110). Modern English Publications in association with The British Council.
   Google Scholar
• Witte, S. P., & Faigley, L. (1981). Coherence, cohesion, and writing quality. College Composition and Communication, 32(2), 189–204. https://doi.org/10.2307/356693
   Google Scholar
• Yang, W., & Sun, Y. (2012). The use of cohesive devices in argumentative writing by Chinese EFL learners at different proficiency levels. Linguistics and Education, 23(1), 31–48. https://doi.org/10.1016/j.linged.2011.09.004
   Google Scholar
• Zemach, D. E., & Stafford-Yilmaz, L. (2008). Writers at work—the essay. Cambridge University Press.
   Google Scholar
• Zhu, W. (2001). Performing argumentative writing in english: Difficulties, processes, and strategies. TESL Canada Journal, 19(1), 1. https://doi.org/10.18806/tesl.v19i1.918
   Google Scholar