Using Mixed Methods to Interpret Differential Item Functioning

ABSTRACT

Analysis of differential item functioning (DIF) is often used to determine if cross-lingual assessments are equivalent across languages. However, evidence on the causes of cross-lingual DIF is still evasive. Expert appraisal is a qualitative method useful for obtaining detailed information about problematic elements in the different linguistic versions of items. In this article we propose and explore a mixed methods approach that integrates quantitative results from DIF analysis and qualitative findings from expert appraisal to discover reasons why items exhibit DIF across languages. First, polytomous DIF was analyzed in responses to the U.S. and Spanish version of scales from the PISA Student Questionnaire by Differential Step Functioning and Ordinal Logistic Regression. Items flagged by both methods were selected to be studied to interpret DIF causes. Secondly, experts were asked about non-comparable elements in items. Experts provided qualitative evidence on problematic issues (different interpretation patterns or response processes), that may have been the cause of the DIF. The integration of results from both methods was aimed at relating type of DIF to expert appraisal findings.