Using Multilevel Modeling in Language Assessment Research: A Conceptual Introduction

Abstract

This article critiques traditional single-level statistical approaches (e.g., multiple regression analysis) to examining relationships between language test scores and variables in the assessment setting. It highlights the conceptual, methodological, and statistical problems associated with these techniques in dealing with multilevel or nested data and discusses an alternative approach, multilevel modeling (MLM), that can handle such data appropriately. An example focusing on contrast effects in essay rating is used to illustrate the main points discussed in the paper and the applications and advantages of MLM. The article also discusses some of the main considerations and issues in MLM (e.g., model building and testing, centering) and concludes by pointing out areas where MLM can be applied in language assessment research.

Notes

¹There are numerous ways to combine analytic scores, depending on assessment purpose and context. The simplest approach is to sum or average them, which means assigning equal weight to all rating criteria. Other approaches involve assigning more weight to some criteria than others based on statistical, practical, and/or theoretical considerations.

²A MANOVA could be used here, but to keep the discussion short and simple, an ANOVA is used.

³In this discussion, holistic scores are used as the outcome, but the same equation applies to the analytic scores as well.

⁴See the discussion of centering that follows.

⁵The following terms are used interchangeably in this article: residual term, error term, unexplained variance, and unmodeled variance.

⁶In an ANOVA, independent variables are always categorical (e.g., group), whereas in MR, independent variables can be either continuous (e.g., test scores) or categorical.

⁷Readers are referred to the references just listed for a statistical explanation of this point.

⁸ Kreft and Leeuw (1998) noted that the sample size lies somewhere between the number of ratings and the number of raters depending on the amount of variance within and between raters (i.e., ICC).

⁹Aggregation may be less common in language assessment research because such research usually includes a small number of observations.

¹⁰ It is possible that these authors considered the issues just discussed, but there is no indication in the publications themselves whether the issue of nested data and its implications were examined empirically (e.g., by computing ICC).

¹¹ ANOVA, G-theory, and MFRM also cannot estimate mediating effects of Level-2 factors (e.g., rater characteristics) on the relationships between Level-1 variables (e.g., between EPL and target essay score).

¹² Context here refers to the context of the ratings that includes the rater, the rating panel (e.g., size and composition of panel), physical context of rating (e.g., place, time), and so forth.

¹³ MLM models fall into two broad statistical categories: a multiple regression approach and a structural equation modeling (SEM) approach (Heck & Thomas, 2000; Hox, 2002). This article adopts a multiple regression approach because it is conceptually less complex.

¹⁴ As is shown next, a first step in conducting MLM is to run a null model to estimate ICC and determine whether one needs to conduct MLM or, if ICC is close to 0, one can use MR.

¹⁵ Level 2 variables can be categorical (e.g., novice vs. experienced) or continuous (e.g., number of years of experience).

¹⁶ The decisions and considerations discussed in this section are similar to those in SEM (see Kunnan, 1998).

¹⁷ Computer programs such as HLM provide user-friendly tools, examples, and step-by-step guidelines for building and evaluating various MLM models.

¹⁸ As Hox (1995) explained, if the models being compared are not nested models, “the principle that models should be as simple as possible (theories and models should be parsimonious) indicates that we should generally stick with the simpler model” (p. 17).

¹⁹ It is the ratio of the true parameter variance to the observed variance (which consists of true and error variances; Deadrick et al., 1997).

²⁰ It should be noted here that conventional statistical models (i.e., ANOVA, MR) form the building blocks of MLM and can be considered special single-level cases of MLM (Hox, 2002).

²¹ Analyses of the residuals from the final model in the Applying MLM to the Data Set section indicated that the distribution of Level-1 and Level-2 residuals did not depart significantly from normality. However, the novice raters tended to have higher residual variances, suggesting more unpredictable variability in the ratings of these raters compared to those of the experienced raters.

²² Some of the scenarios described here can be modeled using G-theory if the predictors are categorical. Where predictors are continuous or mixed, only MLM can handle such data.