Examining the inseparability of content knowledge from LSP reading ability: an approach combining bifactor-multidimensional item response theory and structural equation modeling

ABSTRACT

This study examined the separability of domain-general and domain-specific content knowledge from Language for Specific Purposes (LSP) reading ability. A pool of 1,491 nursing students in China participated by responding to a nursing English test and a nursing knowledge test. Primary data analysis involved four steps: (a) conducting a bifactor-multidimensional item response theory model (bifactor-MIRT) analysis to establish measurement validity for the assumed domain-general factor and domain-specific factors underlying each test and to compute bifactor-MIRT direct scores; (b) transforming the bifactor-MIRT scores into composite scores; (c) conducting a confirmatory factor analysis with the composite scores to reconstruct the orginal bifactor-MIRT models, and (d) conducting a structural equation modeling analysis to explore the relationship between nursing knowledge factors (domain-general and domain-specific) and the nursing English reading factors (domain-general and domain-specific). The results showed that the domain-specific passage factors were significantly correlated with their corresponding domain-specific nursing knowledge factors and that domain-general nursing knowledge significantly predicted the variance of the domain-general reading factor. Overall, we concluded that content knowledge is inseparable from LSP reading ability. The implications for understanding LSP ability and for LSP reading test scoring are discussed.

Notes

¹ A detailed discussion comparing these models is out of the scope of this study. Readers interested in this issue are directed to the work by Steinberg and Thissen (2013).

² To produce stable parameter estimates for a MIRT model, a sample size of 1,000 (Reckase, 2009) to 2,000 (Ackerman, 1994) is necessary. Because the study only had a sample size of 1,491, the present study took the restrictive criteria and constrained the guessing parameter to zero.

³ In statistics, −2LL indicates the difference of model-data fit between a more complex model and a simpler model. A more complex model is preferred when the probability (or p-value) of this difference is significant.

⁴ To avoid information loss, we did not constrain these items simultaneously but by following two steps: First, we only constrained items with relatively large negative loadings (RE5, RE10, and RE13) and found RE8 switched to positive. We then constrained RE1 and RE8 and found RE8 still remained positive. This is possible, given the negative values might result from the test method effect and that, after other items were constrained, the confounding factor was removed, even RE was not constrained.

⁵ We checked the item and found the question asking about the symptom of Migraine, which was not explicitly provided in the text. We concluded that it was mostly due to high demanding on content knowledge that has led to this high discrimination value on the domain factor.

Additional information

Funding

This work was supported by the Educational Testing Service TOEFL Small Grants for Doctoral Research in Second or Foreign Language Assessment; The University of Hong Kong Faculty of Education Research Fund; Assessment Systems Corporation Grants for Graduate Students in Psychological Measurements.