Exploring difficult-to-score essays with a hyperbolic cosine accuracy model and Coh-Metrix indices

Abstract

Unfolding models are frequently used to develop scales for measuring attitudes. Recently, unfolding models have been applied to examine rater severity and accuracy within the context of rater-mediated assessments. One of the problems in applying unfolding models to rater-mediated assessments is that the substantive interpretations of the latent variable are challenging. The purpose of this study is to evaluate the use of Coh-Metrix indices as an approach for defining the characteristics of essays related to their scoring difficulty. Essays from a rater-mediated writing assessment were selected (N = 100), and rater accuracy was defined using an unfolding model. The scoring difficulty of essays along the unfolding scale (i.e., difficult-to-score below, easy-to-score, and difficult-to-score above) was related to essay feature indices defined with the Coh-Metrix text analyzer. Six areas of essay features were considered: descriptive, text readability, lexical diversity, latent semantic analysis, syntactic complexity, and word information. The results suggest that the substantive interpretation of the underlying unfolding scale can be enhanced by examining essay characteristics with Coh-Metrix indices. This methodology offers a promising approach for understanding the unfolding continuum and gaining information about the characteristics of difficult-to-score essays. Suggestions and implications for evaluating rater-mediated assessment systems using unfolding models are discussed.

Keywords:

• Unfolding models
• writing assessments
• rating accuracy
• essay features
• Coh-Metrix indices