Advanced search
Publication Cover
Computer Assisted Language Learning Volume 35, 2022 - Issue 8
Submit an article Journal homepage
5,470
Views
7
CrossRef citations to date
0
Altmetric
Articles

How to provide automated feedback on the writing process? A participatory approach to design writing analytics tools

Rianne Conijna Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands;b Department of Management, University of Antwerp, Antwerp, BelgiumCorrespondence[email protected]
ORCID Iconhttps://orcid.org/0000-0002-6316-4892View further author information
ORCID Icon,
Roberto Martinez-Maldonadoc Faculty of Information Technologies, Monash University, Melbourne, AustraliaORCID Iconhttps://orcid.org/0000-0002-8375-1816View further author information
ORCID Icon,
Simon Knightd Faculty of Transdisciplinary Innovation, University of Technology, Sydney, AustraliaORCID Iconhttps://orcid.org/0000-0002-8709-5780View further author information
ORCID Icon,
Simon Buckingham Shume Connected Intelligence Centre, University of Technology, Sydney, AustraliaORCID Iconhttps://orcid.org/0000-0002-6334-7429View further author information
ORCID Icon,
Luuk Van Waesb Department of Management, University of Antwerp, Antwerp, BelgiumORCID Iconhttps://orcid.org/0000-0002-3642-9533View further author information
ORCID Icon &
Menno van Zaanenf South African Centre for Digital Language Resources, North-West University, Potchefstroom, South AfricaORCID Iconhttps://orcid.org/0000-0003-1841-2444View further author information
ORCID Icon
Pages 1838-1868 | Published online: 10 Nov 2020

References

  • Alamargot, D., & Chanquoy, L. (2001). Through the models of writing (Vol. 9). Dordrecht: Springer. doi:10.1007/978-94-010-0804-4
  • Allen, L. K., Jacovina, M. E., Dascalu, M., Roscoe, R. D., Kent, K., Likens, A. D., & McNamara, D. S. (2016). {ENTER}ing the Time Series {SPACE}: Uncovering the writing process through keystroke analyses. Paper presented at the 9th International Conference on Educational Data Mining (EDM) (pp. 22–29), Raleigh, NC. https://eric.ed.gov/?id=ED592674.
  • Allen, L. K., Jacovina, M. E., & McNamara, D. S. (2015). Computer-based writing instruction. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of Writing Research (pp. 316–329). New York, NY: The Guilford Press Publisher. Retrieved from https://eric.ed.gov/?id=ED586512.
  • Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3), 1–21. doi:10.1002/j.2333-8504.2004.tb01972.x
  • Baaijen, V. M., & Galbraith, D. (2018). Discovery through writing: Relationships with writing processes and text quality. Cognition and Instruction, 36(3), 199–125. doi:10.1080/07370008.2018.1456431.
  • Baaijen, V. M., Galbraith, D., & de Glopper, K. (2012). Keystroke analysis: Reflections on procedures and measures. Written Communication, 29(3), 246–277. doi:10.1177/0741088312451108.
  • Becker, A. (2006). A review of writing model research based on cognitive processes. In A. Horning & A. Becker (Eds.), Revision: History, theory, and practice (pp. 25–49). West Lafayette, IN: Parlor Press. Retrieved from https://wac.colostate.edu/books/horning_revision/chapter3.pdf
  • Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
  • Bixler, R., & D’Mello, S. (2013). Detecting boredom and engagement during writing with keystroke analysis, task appraisals, and stable traits. Paper presented at the 2013 International Conference on Intelligent User Interfaces (pp. 225–234), Santa Monica, CA.   doi:10.1145/2449396.2449426
  • Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi:10.1191/1478088706qp063oa.
  • Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. doi:10.18608/jla.2019.62.1.
  • Buckingham Shum, S., Knight, S., McNamara, D., Allen, L., Bektik, D., & Crossley, S. (2016). Critical perspectives on writing analytics. Paper presented at the Sixth International Conference on Learning Analytics & Knowledge (pp. 481–483), New York. doi:10.1145/2883851.2883854
  • Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in writing: Generating text in L1 and L2. Written Communication, 18(1), 80–98. doi:10.1177/0741088301018001004.
  • Chenoweth, N. A., & Hayes, J. R. (2003). The inner voice in writing. Written Communication, 20(1), 99–118. doi:10.1177/0741088303253572.
  • Conde, M. Á., & Hernández-García, Á. (2015). Learning analytics for educational decision making. Computers in Human Behavior, 47, 1–3. doi:10.1016/j.chb.2014.12.034.
  • Cotos, E. (2015). Automated Writing Analysis for writing pedagogy: From healthy tension to tangible prospects. Writing & Pedagogy, 7(2–3), 197–231. doi:10.1558/wap.v7i2-3.26381.
  • Crossley, S. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415–443. doi:10.17239/jowr-2020.11.03.01.
  • Deane, P. (2014). Using writing process and product features to assess writing quality and explore how those features relate to other literacy tasks. ETS Research Report Series, 2014(1), 1–23. doi:10.1002/ets2.12002.
  • Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment, 5(1). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1640
  • Dollinger, M., Liu, D., Arthars, N., & Lodge, J. (2019). Working together in learning analytics towards the co-creation of value. Journal of Learning Analytics, 6(2), 10–26. doi:10.18608/jla.2019.62.2.
  • El Ebyary, K., & Windeatt, S. (2010). The impact of computer-based feedback on students’ written work. International Journal of English Studies, 10(2), 121–142. doi:10.6018/ijes/2010/2/119231.
  • Ferris, D. (2011). Responding to student errors: Issues and strategies. In D. Ferris (Ed.), Treatment of error in second language student writing (pp. 49–76). Ann Arbor, MI: University of Michigan Press.
  • Flower, L., & Hayes, J. R. (1980). The cognition of discovery: Defining a rhetorical problem. College Composition and Communication, 31(1), 21–32. doi:10.2307/356630.
  • Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College Composition and Communication, 32(4), 365–387. http://www.jstor.org/stable/356600.
  • Gabriska, D., & Ölveckỳ, M. (2018). Issues of adaptive interfaces and their use in educational systems. Paper presented at the 16th International Conference on Emerging eLearning Technologies and Applications (ICETA) (pp. 173–178), Stary Smokovec, Slovakia. doi:10.1109/ICETA.2018.8572096
  • Galbraith, D. (2009). Writing about what we know: Generating ideas in writing. In R. Beard, D. Myhill, J. Riley, & M. Nystrand (Eds.), The SAGE handbook of writing development (pp. 48–64). London, UK: Sage. doi:10.4135/9780857021069.n4
  • Galbraith, D., & Baaijen, V. M. (2019). Aligning keystrokes with cognitive processes in writing. In Observing writing (pp. 306–325). Leiden, Netherlands: Brill.
  • Giacomin, J. (2014). What is human centred design? The Design Journal, 17(4), 606–623. doi:10.2752/175630614X14056185480186
  • Graham, S., McKeown, D., Kiuhara, S., & Harris, K. R. (2012). A meta-analysis of writing instruction for students in the elementary grades. Journal of Educational Psychology, 104(4), 879–896. doi:10.1037/a0029185
  • Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99(3), 445–476. doi:10.1037/0022-0663.99.3.445
  • Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing. In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 1–27). NJ: Erlbaum.
  • Hayes, J. R. (2012). Modeling and remodeling writing. Written Communication, 29(3), 369–388. doi:10.1177/0741088312451260
  • Hounsell, D. (1984). Essay planning and essay writing. Higher Education Research and Development, 3(1), 13–31. doi:10.1080/0729436840030102
  • Hounsell, D. (1997). Contrasting conceptions of essay-writing. The Experience of Learning, 2, 106–125.
  • Itua, I., Coffey, M., Merryweather, D., Norton, L., & Foxcroft, A. (2014). Exploring barriers and solutions to academic writing: Perspectives from students, higher education and further education tutors. Journal of Further and Higher Education, 38(3), 305–326. doi:10.1080/0309877X.2012.726966
  • Kaufer, D. S., Hayes, J. R., & Flower, L. (1986). Composing written sentences. Research in the Teaching of English, 20(2), 121–140. Retrieved from http://www.jstor.org/stable/40171073
  • Kellogg, R. T. (1996). A model of working memory in writing. In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 57–71). NJ: Erlbaum.
  • Kidd, P. S., & Parshall, M. B. (2000). Getting the focus and the group: Enhancing analytical rigor in focus group research. Qualitative Health Research, 10(3), 293–308. doi:10.1177/104973200129118453
  • Knight, S., Martinez-Maldonado, R., Gibson, A., & Buckingham Shum, S. (2017). Towards mining sequences and dispersion of rhetorical moves in student written texts. Paper presented at the Seventh International Learning Analytics & Knowledge Conference (pp. 228–232), Vancouver, Canada.   doi:10.1145/3027385.3027433
  • Knight, S., Shibani, A., Abel, S., Gibson, A., & Ryan, P. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186. doi:10.17239/jowr-2020.12.01.06.
  • Lea, M. R., & Street, B. V. (1998). Student writing in higher education: An academic literacies approach. Studies in Higher Education, 23(2), 157–172. doi:10.1080/03075079812331380364.
  • Leijten, M., & Van Waes, L. (2013). Keystroke logging in writing research: Using Inputlog to analyze and visualize writing processes. Written Communication, 30(3), 358–392. doi:10.1177/0741088313491692.
  • Leijten, M., Van Waes, L., Schriver, K., & Hayes, J. R. (2014). Writing in the workplace: Constructing documents using multiple digital sources. Journal of Writing Research, 5(3), 285–337. doi:10.17239/jowr-2014.05.03.3.
  • Levy, M., & Moore, P. J. (2018). Qualitative research in CALL. Language Learning & Technology, 22(2), 1–7. https://www.lltjournal.org/item/10125-44638/.
  • Likens, A. D., Allen, L. K., & McNamara, D. S. (2017). Keystroke dynamics predict essay quality. Paper presented at the 39th Annual Meeting of the Cognitive Science Society (CogSci 2017) (pp. 2573–2578), London.
  • Limpo, T., Nunes, A., & Coelho, A. (2020). Introduction to the special issue on technology-based writing instruction: A collection of effective tools. Journal of Writing Research, 12(1), 1–7. doi:10.17239/jowr-2020.12.01.01.
  • Lindgren, E., & Sullivan, K. P. (2019). Observing writing: Insights from keystroke logging and handwriting. Leiden, Netherlands: Brill. doi:10.1163/9789004392526
  • Link, S., Dursun, A., Karakaya, K., & Hegelheimer, V. (2014). Towards best ESL practices for implementing automated writing evaluation. Calico Journal, 31(3), 323–344. https://www.jstor.org/stable/calicojournal.31.3.323.
  • Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. doi:10.1177/0002764213479367.
  • Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901–918. doi:10.1037/a0037123.
  • Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A. (2016). LATUX: An iterative workflow for designing, validating and deploying learning analytics visualisations. Journal of Learning Analytics, 2(3), 9–39. doi:10.18608/jla.2015.23.3.
  • Medimorec, S., & Risko, E. F. (2017). Pauses in written composition: On the importance of where writers pause. Reading and Writing, 30(6), 1267–1285. doi:10.1007/s11145-017-9723-7.
  • Norton, L. S. (1990). Essay-writing: What really counts? Higher Education, 20(4), 411–442. doi:10.1007/BF00136221.
  • NVivo. (2015). NVivo: NVivo qualitative data analysis software (Version 11). [Computer software]. Doncaster, Australia: QSR International Pty Ltd.
  • Prieto-Alvarez, C. G., Martinez-Maldonado, R., & Anderson, T. (2018). Co-designing learning analytics tools with learners. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers (pp. 93–110). London: Routledge.
  • Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653–674. doi:10.1080/09588221.2018.1428994.
  • Ranalli, J., Feng, H.-H., & Chukharev-Hudilainen, E. (2018). The affordances of process-tracing technologies for supporting L2 writing instruction. Language Learning & Technology, 23(2), 1–11. https://scholarspace.manoa.hawaii.edu/items/236c5a21-a813-4594-943c-b09cb0272292.
  • Rapp, C., & Kauf, P. (2018). Scaling academic writing instruction: Evaluation of a scaffolding tool (Thesis Writer). International Journal of Artificial Intelligence in Education, 28(4), 590–615. doi:10.1007/s40593-017-0162-z.
  • Rijlaarsdam, G., & Van den Bergh, H. (1996). The dynamics of composing – An agenda for research into an interactive compensatory model of writing: Many questions, some answers. In C. M. Levy & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 107–125). NJ: Lawrence Erlbaum Associates, Inc.
  • Roscoe, R. D., Allen, L. K., Weston, J. L., Crossley, S. A., & McNamara, D. S. (2014). The Writing Pal intelligent tutoring system: Usability testing and development. Computers and Composition, 34, 39–59. doi:10.1016/j.compcom.2014.09.002.
  • Shibani, A., Knight, S., & Shum, S. B. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. Paper presented at the 9th International Conference on Learning Analytics & Knowledge (pp. 210–219), New York, NY. doi:10.1145/3303772.3303785.
  • Solé, I., Miras, M., Castells, N., Espino, S., & Minguela, M. (2013). Integrating information: An analysis of the processes involved and the products generated in a written synthesis task. Written Communication, 30(1), 63–90. doi:10.1177/0741088312466532.
  • Staples, S., Egbert, J., Biber, D., & Gray, B. (2016). Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre. Written Communication, 33(2), 149–183. doi:10.1177/0741088316631527.
  • Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347. doi:10.1037/a0034752.
  • Teasley, S. D. (2017). Student facing dashboards: One size fits all? Technology, Knowledge and Learning, 22(3), 377–384. doi:10.1007/s10758-017-9314-3.
  • Tillema, M., van den Bergh, H., Rijlaarsdam, G., & Sanders, T. (2011). Relating self reports of writing behaviour and online task execution using a temporal model. Metacognition and Learning, 6(3), 229–253. doi:10.1007/s11409-011-9072-x.
  • Van Lier, L. (2000). From input to affordance: Social-interactive learning from an ecological perspective. In J. Lantolf (Ed.), Sociocultural theory and second language learning (Vol. 78, pp. 245–259). Oxford: Oxford University Press.
  • Van Waes, L., van Weijen, D., & Leijten, M. (2014). Learning to write in an online writing center: The effect of learning styles on the writing process. Computers & Education, 73, 60–71. doi:10.1016/j.compedu.2013.12.009.
  • Vandermeulen, N., Leijten, M., & Van Waes, L. (2020). Reporting writing process feedback in the classroom: Using keystroke logging data to reflect on writing processes. Journal of Writing Research, 12(1), 109–140. doi:10.17239/jowr-2020.12.01.05.
  • Wallot, S., & Grabowski, J. (2013). Typewriting dynamics: What distinguishes simple from complex writing tasks? Ecological Psychology, 25(3), 267–280. doi:10.1080/10407413.2013.810512.
  • Wang, Y.-J., Shang, H.-F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 234–257. doi:10.1080/09588221.2012.655300.
  • Wengelin, Å. (2006). Examining pauses in writing: Theory, methods and empirical data. In K. P. H. Sullivan & E. Lindgren (Ed.), Computer key-stroke logging and writing: Methods and applications (Studies in Writing) (Vol. 18, pp. 107–130). Oxford: Elsevier.
  • Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. doi:10.18608/jla.2019.62.4.
  • Wolsey, T. D., Lapp, D., & Fisher, D. (2012). Students’ and teachers’ perceptions: An inquiry into academic writing. Journal of Adolescent & Adult Literacy, 55(8), 714–724. doi:10.1002/JAAL.00086.
  • Woolner, P., Hall, E., Wall, K., & Dennison, D. (2007). Getting together to improve the school environment: User consultation, participatory design and student voice. Improving Schools, 10(3), 233–248. doi:10.1177/1365480207077846.
  • Xu, C. (2018). Understanding online revisions in L2 writing: A computer keystroke-log perspective. System, 78, 104–114. doi:10.1016/j.system.2018.08.007.
  • Zhang, F., & Litman, D. (2015). Annotation and classification of argumentative writing revisions. Paper presented at the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 133–143), Denver, COL. doi:10.3115/v1/W15-0616
  • Zhang, M., Hao, J., Li, C., & Deane, P. (2016). Classification of writing patterns using keystroke logs. In L. A. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, & M. Wiberg (Eds.), Quantitative psychology research: The 80th annual meeting of the Psychometric Society, Beijing, 2015 (pp. 299–314). Beijing, China: Springer. doi:10.1007/978-3-319-38759-8_23.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.