Method Effects: The Problem With Negatively Versus Positively Keyed Items

Abstract

Using confirmatory factor analyses, we examined method effects on Rosenberg's Self-Esteem Scale (RSES; Rosenberg, 1965) in a sample of older European adults. Nine hundred forty nine community-dwelling adults 60 years of age or older from 5 European countries completed the RSES as well as measures of depression and life satisfaction. The 2 models that had an acceptable fit with the data included method effects. The method effects were associated with both positively and negatively worded items. Method effects models were invariant across gender and age, but not across countries. Both depression and life satisfaction predicted method effects. Individuals with higher depression scores and lower life satisfaction scores were more likely to endorse negatively phrased items.

Notes

There were significant differences across countries in terms of age, F(4, 933) = 6.52, p < .001. Participants from the United Kingdom were the oldest (M = 75.61, SD = 7.71) and participants from Italy were the youngest  (M = 72.48, SD = 7.71). There were also differences in terms of the gender distribution across countries, χ²(4, N = 949) = 64.77, p < .001. There were a larger proportion of women in the British and Finnish samples (75%) compared with the Swedish (65%), Italian (60%) and Greek (42%) samples. Moreover, there were also differences in terms of highest level of education, χ²(8, N = 932) = 287.99, p < .001. The largest proportion in the U.K. sample reported tertiary education as their highest level of education. In the Swedish and Finnish samples, however, the largest proportion reported secondary education, and in the Italian and Greek samples, most participants reported primary education as their highest level. Finally, there were significant differences in self-esteem scores across countries, F(4, 933) = 6.52, p < .001. The British sample reported significantly (ps < .05) higher self-esteem scores (M = 2.55, SD = .22) than the Italian (M = 2.48, SD = .26), Swedish (M = 2.46, SD = .26), Finnish (M = 2.30, SD = .25) and Greek (M = 2.25, SD = .28) samples, and the Greek and Finnish samples reported significantly (ps < .001) lower scores than the other countries.

We found Model 5 to be empirically underidentified. Hence, similar to the procedure in previous studies (e.g., Tomas & Oliver, 1999) we constrained the correlated uniqueness between the errors of Items 3 and 8 to 0 as this correlation was nonsignificant.

The complete results from all the invariance analyses are available from the first author on request.

As the Swedish sample only included 47 participants, we choose not to use this sample in the invariance analyses across countries. Thus, the invariance analyses across countries included the four samples from Finland, Italy, Greece, and the United Kingdom.