An Innovative Approach to Assessing Depth of Knowledge of Academic Words

ABSTRACT

The purpose of this research is to develop a vocabulary assessment, Evaluation of Academic Vocabulary or “EAV,” that gauges students’ depth of knowledge of academic words and is sensitive to vocabulary growth in the context of literacy interventions. We report results from two studies with native English-speaking middle school students designed to inform the development and pilot test the utility and technical qualities of an innovative vocabulary assessment that is sensitive to students’ level of understanding of general academic words. Across the two studies, we employed a mixed methods research approach. In the first study, we drew on classic psychometric assessment methods, Bayesian network methods, signal detection theory to evaluate the validity and technical qualities of the assessment. In the latter study, we conducted cognitive interviews to further validate the assessment. Results suggest that: 1) the assessment captures growth in knowledge of general academic words, i.e., words that are important for the comprehension of academic texts; and 2) it is sensitive to treatment effects academic vocabulary intervention for adolescents.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ Since publication of the AWL, newer word lists have become available based on rigorous methodologies and larger corpora such as the Academic Vocabulary List (AVL) (Gardner & Davies, 2013) and the New Academic Word List (NAWL) (Browne, Culligan, & Phillips, 2013). For our purposes, AWL was sufficient for identifying target words that are dispersed with frequency across content domains in middle school texts.

² While we believe that application of Bayes’ rule produced more precise estimates of word knowledge, it is also possible that a student could have arrived at either of these patterns as a result of guessing by applying a blanket “accept all” or “reject all” policy. This possibility was one of the motivations for Study Two in which we carried out cognitive interviews to understand the strategies and reasoning employed by students to make decisions on this assessment.

Additional information

Funding

This work was supported by the Institute of Education Sciences [Award R305A100440].