Using Topic Models to Understand Rater-Mediated Writing Assessments

Abstract

Integrated writing assessments are tasks that require students to construct an essay using evidence from source texts. Topic models are statistical models used to analyze text. This study analyzes two integrated writing tasks from students in grades 6 (N=1,250) and 8 (N=1,250) using topic models. Independent evaluations of the essays were also conducted to help generalize the results across the two grades and better understand students' underlying writing techniques. Four main categories were identified: (1) use of words that appeared in the prompt, (2) words that appeared in the source texts, (3) words that reflected inference, and (4) words that reflected integration. Higher-scoring students used words that reflect inference and integration of source text more often than lower-performing students.

Keywords:

• Integrated writing assessments
• latent Dirichlet allocation
• rater-mediated assessments
• topic models

Notes

1 The middle quality essay groups (essays scored with a 4 out of a 7 point scale) were not further investigated beyond Figure 2 due to the fact that major differences between essays were seen between the low and high quality essay groups.

2 As with the sixth grade analysis, the middle quality essay groups (essays scored with a 4 out of a 7 point scale) were not further investigated beyond Figure 2 due to the fact that major differences between essays were seen between the low and high quality essay groups

3 It is important to note that Topic 3 from the eighth grade prompt is different from Topic 3 in the sixth grade prompt. The numbering of each topic is simply a nominal (or categorical) designation determined during the estimation procedure and is not related to the content of the topic.