Understanding Learner Strengths and Weaknesses: Assessing Performance on an Integrated Writing Task

Abstract

The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT® (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors—Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation.

ACKNOWLEDGMENTS

This study was funded by Educational Testing Service. The authors are indebted to the assessment development specialists at ETS and ESL practitioners who participated in the analytic scoring scheme development and the actual scoring work. Our special thanks go as well to Cathy Trapani and Vincent Weng for extracting the TOEFL examinee performance data for this study; Robert Kantor and Jakub Novak for rating rubric development and rater training for analytic scoring; Tom Florek and Diana X. Wang for creating rating software; and Yeonsuk Cho for reliability analyses of rating data.

Notes

¹TOEFL iBT scores set by universities and other institutions (http://www.ets.org/toefl/institutions/scores/use/ibt_score_setting).