Are the indicators for the Language and Reasoning Subscale of the Early Childhood Environment Rating Scale‐Revised psychometrically appropriate for Caribbean classrooms?

Abstract

Evaluating the psychometric properties of the indicators that comprise the Early Childhood Environment Rating Scale‐Revised (ECERS‐R) language‐reasoning scale from an item response theory (IRT) perspective on a sample of observations from 334 Caribbean classrooms, Stout’s procedure revealed that all indicators on this dimension are not part of a single essentially unidimensional construct. IRT‐based factor analyses on the indicator scores yielded two factors – named Language‐Reasoning Activities and Language‐Reasoning Materials. IRT analyses conducted on these two factors revealed that their indicators provide adequate psychometric information and have no floor effects – although they demonstrate evidence for ceiling effects. IRT also revealed that at least within the Caribbean context: (a) the ECERS‐R authors have ordered the indicators inappropriately; (b) administration of all indicators is unnecessary; and (c) equally weighting indicators might yield spurious results. IRT‐based scoring might improve the psychometric soundness of indicators on this ECERS‐R scale.

Keywords:

• Caribbean
• early childhood classrooms
• ECERS‐R
• item response theory

Notes

1. It is important to note that reference to items on the ECERS‐R is reflective of 43 items. The scores for each of these items are, however, derived from scores on multiple dichotomously scored indicators (see description of the ECERS‐R under the ‘Measure’ subheading).

2. Our use of indicators is not the typical description used in psychometric theory, but these terms are taken from the ECERS‐R and describe the dichotomously scored items from which each of the 43 items is scored.

3. Other methods such as weighted least squares with tetrachoric correlations have been demonstrated to provide similar results to FIFA and can assess data‐to‐model fit with goodness‐of‐fit indices (see Woods 2002).

4. TESTFACT allows the researcher to use the marginal maximum likelihood estimate in a bfactor solution that is nested in a previously selected FIFA factor solution and permits the use of a likelihood ratio test for conditional independence. A significantly lower likelihood ratio estimate for a bi‐factor solution would suggest the presence of a primary (e.g. second‐order) factor. The presence of a second‐order factor would provide evidence that conditional independence is questionable and that a second‐order factor is necessary to explain the data. In other words, a relationship between indicators would exist even when the trait level is held constant (Panter and Reeve 2002).

5. The likelihood ratio test suggested a three‐factor model. That is, the FIFAs where different factor models were compared starting from a one‐ and a two‐factor model revealed that the three‐ and four‐factor models were the first pair of analyses to reveal no significant differences, Δχ² (25) = 12.92, p > .96. The third‐factor in the three‐factor solution had loadings accounting for less than 1% of the variance. Furthermore, the three‐factor model had multiple cross‐loaded indicators, thereby raising concerns regarding the robustness of these factors. The two‐factor model was theoretically plausible, since the first factor represented activities that facilitate language and reasoning and the second factor materials used to promote language and reasoning. Procedures Gibbons and Hedeker (1992) used in selecting a factor model based on dominant indicator to factor loadings were therefore used in selecting the two‐factor solution used in addressing the next IRT assumption.

6. TESTFACT was used to test for conditional independence by nesting a bi‐factor model in the two‐factor model. Comparing likelihood ratios across the nested models revealed a nonsignificant effect, Δχ² (3) = 5.77, p >.10, and showed no significant difference between the bi‐factor likelihood ratio and that of a simpler factor structure. This finding suggests that the two‐factor model might be appropriately specified and that a primary dimension was not necessary to fully describe the data.

7. For each dimension, the MULTILOG analyses used to test whether 1‐, 2‐, or 3‐parameter models best represented responses were conducted in models where each indicator was unconstrained across the two nations. In the first test, the 1PL model was nested in the 2PL model and in the second test the 2PL model was nested in the 3PL model. For the Language and Reasoning Activities factor the comparisons between the 2PL and the 1PL model showed better fit for the 2PL model, since its G² was significantly lower than the G² for the 1PL model, ΔG² (12) = 72.3, p < .001. Comparing the 2PL model with the 3PL model showed no better fit for the 2PL model, since its G² was not significantly different from the G² for the 3PL model, ΔG² (108) = 55.8 p >.99, suggesting no significant decrement in the G² statistic when 3PL model was applied. The 2PL model was therefore chosen to estimate the parameters for the indicators. Similar trends emerged between comparisons of the 1PL and 2PL models for the Language and Reasoning Materials factor, Δχ² (9) =29.4,1. p < .001. With Δχ2 (10) = 12.4, p >.26 for the 2PL versus the 3PL models. There was no significant difference between the models, suggesting that no significant improvement emerged when the less parsimonious 3PL model was applied.

8. We note that Factors 1 and 2 include 13 and 10 indicators, respectively. Nevertheless, to present uncluttered and legible figures, only four indicators for each factor are selected for illustration.