Examining and Controlling for Wording Effect in a Self-Report Measure: A Monte Carlo Simulation Study

Abstract

Wording effect refers to the systematic method variance caused by positive and negative item wordings on a self-report measure. This Monte Carlo simulation study investigated the impact of ignoring wording effect on the reliability and validity estimates of a self-report measure. Four factors were considered in the simulation design: (a) the number of positively and negatively worded items, (b) the loadings on the trait and the wording effect factors, (c) sample size, and (d) the magnitude of population validity coefficient. The findings suggest that the unidimensional model that ignores the negative wording effect would underestimate the composite reliability and criterion-related validity, but overestimate the homogeneity coefficient. The magnitude of relative bias of the composite reliability was generally small and acceptable, whereas the relative bias for the homogeneity coefficient and criterion-related validity coefficient was negatively correlated with the strength of the general trait factor.

Keywords:

• bifactor model
• Monte Carlo simulation
• reliability
• validity
• wording effect

FUNDING

This research was funded by grants from the National Natural Science Foundation of China (31271116, 31400909).

Additional information

Funding

This research was funded by grants from the National Natural Science Foundation of China (31271116, 31400909).