Evaluating Writing Process Features in an Adult EFL Writing Assessment Context: A Keystroke Logging Study

ABSTRACT

Keystroke logs provide a comprehensive record of observable writing processes. Previous studies examining the keystroke logs of young L1 English writers performing experimental writing tasks have identified writing processes features predictive of the quality of responses. Contrarily, large-scale studies on the dynamic and temporal nature of L2 writing process are scarce, especially in an assessment setting. This study utilized the keystroke logs of adult English as a foreign language (EFL) learners responding to assessment tasks to examine the usefulness of the process features in this new context. We evaluated the features in terms of stability, explored factor structures for their correlations, and constructed models to predict response quality. The results showed that most of the process features were stable and that their correlations could be efficiently represented with a five-factor structure. Moreover, we observed improved response quality prediction over a baseline by up to 48%. These findings have implications for the evaluation and understanding of writing process features and for the substantive understanding of writing processes under assessment conditions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ This approach treats the total word count of a given response as fixed and measures the time to reach the milestones. Another possible approach is to treat the total writing time as fixed and to measure the word counts at certain time points (see, e.g., Barkaoui, 2016).

² We tracked the final 200 keystrokes because they represented approximately the final 10% of the average total keystrokes (see Table A in Appendix).

³ Another possibility was to build models with the factor scores from the earlier EFA models instead of individual features. However, factor scores from EFA models have a well-known issue of indeterminacy and their use is controversial (for a review, see Mauran, 1996). Moreover, since we built separate models for the task types, the resulting factor scores would have been calculated with different weights across the task types. Because of these reasons, we decided not to use the factor scores for our predictive models.

⁴ We also examined the factor loading and correlation matrices estimated by the least squares method, which does not require the normality assumption. The least squares estimates showed practically identical patterns to those in Tables 4 and 5.

⁵ This label is for convenience. As noted by Abdel Latif (2013), writing fluency has been defined differently by different researchers. It is not our intention to propose a definition of writing fluency.