Examining and Controlling for Wording Effect in a Self-Report Measure: A Monte Carlo Simulation Study

REFERENCES

• Alessandri, G., Vecchione, M., Fagnani, C., Bentler, P. M., Barbaranelli, C., Medda, E., … Caprara, G. V. (2010). Much more than model fitting? Evidence for the heritability of method effect associated with positively worded items of the Life Orientation Test Revised. Structural Equation Modeling, 17, 642–653. doi:10.1080/10705511.2010.510064
   Web of Science ®Google Scholar
• Bandalos, D. L. (2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9, 78–102. doi:10.1207/S15328007SEM0901_5
   Web of Science ®Google Scholar
• Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137–143. doi:10.1007/s11336-008-9100-1
   PubMed Web of Science ®Google Scholar
• Bradley, J. V. (1978). Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144–152. doi:10.1111/bmsp.1978.31.issue-2
   Web of Science ®Google Scholar
• Brunner, M., & Süb, H.-M. (2005). Analyzing the reliability of multidimensional measures: An example from intelligence research. Educational and Psychological Measurement, 65, 227–240. doi:10.1177/0013164404268669
   Web of Science ®Google Scholar
• Chen, F. F., West, S. G., & Sousa, K. H. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189–225. doi:10.1207/s15327906mbr4102_5
   PubMed Web of Science ®Google Scholar
• Dalal, D. K., & Carter, N. T. (2015). Negatively worded items negatively impact survey research. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends (pp. 112–132). New York, NY: Routledge.
   Google Scholar
• DiStefano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13, 440–464. doi:10.1207/s15328007sem1303_6
   Web of Science ®Google Scholar
• DiStefano, C., & Motl, R. W. (2009). Personality correlates of method effects due to negatively worded items on the Rosenberg Self-Esteem Scale. Personality and Individual Differences, 46, 309–313. doi:10.1016/j.paid.2008.10.020
   Web of Science ®Google Scholar
• Donnellan, M. B., Ackerman, R. A., & Brecheen, C. (2016). Extending structural analyses of the Rosenberg Self-Esteem Scale to consider criterion-related validity: Can composite self-esteem scores be good enough? Journal of Personality Assessment, 98, 169–177. doi:10.1080/00223891.2015.1058268
   PubMed Web of Science ®Google Scholar
• Ebesutani, C., Drescher, C. F., Reise, S. P., Heiden, L., Hight, T. L., & Young, J. (2012). The importance of modeling method effects: Resolving the (uni)dimensionality of the Loneliness Questionnaire. Journal of Personality Assessment, 94, 186–195. doi:10.1080/00223891.2011.627967
   PubMed Web of Science ®Google Scholar
• Gana, K., Saada, Y., Bailly, N., Joulain, M., Hervé, C., & Alaphilippe, D. (2013). Longitudinal factorial invariance of the Rosenberg Self-Esteem Scale: Determining the nature of method effects due to item wording. Journal of Research in Personality, 47, 406–416. doi:10.1016/j.jrp.2013.03.011
   Web of Science ®Google Scholar
• Gignac, G. (2010). Seven-factor model of emotional intelligence as measured by Genos EI. European Journal of Psychological Assessment, 26, 309–316. doi:10.1027/1015-5759/a000041
   Web of Science ®Google Scholar
• Gignac, G. E., Palmer, B. R., & Stough, C. (2007). A confirmatory factor analytic investigation of the TAS–20: Corroboration of a five-factor model and suggestions for improvement. Journal of Personality Assessment, 89, 247–257. doi:10.1080/00223890701629730
   PubMed Web of Science ®Google Scholar
• Gu, H., Wen, Z., & Fan, X. (2015). The impact of wording effect on reliability and validity of the Core Self-Evaluation Scale (CSES): A bi-factor perspective. Personality and Individual Differences, 83, 142–147. doi:10.1016/j.paid.2015.04.006
   Web of Science ®Google Scholar
• Gu, H., Wen, Z., & Fan, X. (2017). Structural validity of the Machiavellian Personality Scale: A bifactor exploratory structural equation modeling approach. Personality and Individual Differences, 105, 116–123. doi:10.1016/j.paid.2016.09.042
   Web of Science ®Google Scholar
• Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods & Research, 26, 329–368. doi:10.1177/0049124198026003003
   Web of Science ®Google Scholar
• Lindwall, M., Barkoukis, V., Grano, G., Lucidi, F., Raudsepp, L., Liukkonen, J., & Thøgersen-Ntoumani, C. (2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94, 196–204. doi:10.1080/00223891.2011.645936
   PubMed Web of Science ®Google Scholar
• Marsh, H. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70, 810–819. doi:10.1037/0022-3514.70.4.810
   PubMed Web of Science ®Google Scholar
• Marsh, H. W., Scalas, L. F., & Nagengast, B. (2010). Longitudinal tests of competing factor structures for the Rosenberg Self-Esteem Scale: Traits, ephemeral artifacts, and stable response styles. Psychological Assessment, 22, 366–381. doi:10.1037/a0019225
   PubMed Web of Science ®Google Scholar
• McKay, M. T., Boduszek, D., & Harvey, S. A. (2014). The Rosenberg Self-Esteem scale: A bifactor answer to a two-factor question? Journal of Personality Assessment, 96, 654–660. doi:10.1080/00223891.2014.923436
   PubMed Web of Science ®Google Scholar
• McLarnon, M. J., Goffin, R. D., Schneider, T. J., & Johnston, N. G. (2016). To be or not to be: Exploring the nature of positively and negatively keyed personality items in high-stakes testing. Journal of Personality Assessment, 98, 480–490. doi:10.1080/00223891.2016.1170691
   PubMed Web of Science ®Google Scholar
• Michaelides, M. P., Koutsogiorgi, C., & Panayiotou, G. (2016). Method effects on an adaptation of the Rosenberg Self-Esteem Scale in Greek and the role of personality traits. Journal of Personality Assessment, 98, 178–188. doi:10.1080/00223891.2015.1089248
   PubMed Web of Science ®Google Scholar
• Morin, A. J. S., Arens, A. K., & Marsh, H. W. (2016). A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Structural Equation Modeling, 23, 116–239. doi:10.1080/10705511.2014.961800
   Web of Science ®Google Scholar
• Motl, R. W., Conroy, D. E., & Horan, P. M. (2000). The Social Physique Anxiety Scale: An example of the potential consequences of negatively worded items in factorial validity studies. Journal of Applied Measurement, 1, 327–345.
   PubMedGoogle Scholar
• Motl, R. W., & DiStefano, C. (2002). Longitudinal invariance of self-esteem and method effects associated with negatively worded items. Structural Equation Modeling, 9, 562–578. doi:10.1207/S15328007SEM0904_6
   Web of Science ®Google Scholar
• Paiva-Salisbury, M. L., Gill, A. D., & Stickle, T. R. (2016). Isolating trait and method variance in the measurement of callous and unemotional traits. Assessment Advanced online publication. doi:10.1177/1073191115624546
   PubMedGoogle Scholar
• Quilty, L. C., Oakman, J. M., & Risko, E. (2006). Correlates of the Rosenberg Self-Esteem Scale method effects. Structural Equation Modeling, 13, 99–117. doi:10.1207/s15328007sem1301_5
   Web of Science ®Google Scholar
• Ray, J. V., Frick, P. J., Thornton, L. C., Steinberg, L., & Cauffman, E. (2016). Positive and negative item wording and its influence on the assessment of callous-unemotional traits. Psychological Assessment, 28, 394–404. doi:10.1037/pas0000183
   PubMed Web of Science ®Google Scholar
• Raykov, T., & Grayson, D. (2003). A test for change of composite reliability in scale development. Multivariate Behavioral Research, 38, 143–159. doi:10.1207/S15327906MBR3802_1
   Web of Science ®Google Scholar
• Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667–696. doi:10.1080/00273171.2012.715555
   PubMed Web of Science ®Google Scholar
• Reise, S. P., Scheines, R., Widaman, K. F., & Haviland, M. G. (2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 1, 5–26. doi:10.1177/0013164412449831
   Web of Science ®Google Scholar
• Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145–154. doi:10.1007/s11336-008-9102-z
   Web of Science ®Google Scholar
• Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 98, 223–237.
   Google Scholar
• Schwartz, S. J., Zamboanga, B. L., Wang, W., & Olthuis, J. V. (2009). Measuring identity from an Eriksonian perspective: Two sides of the same coin? Journal of Personality Assessment, 91, 143–154. doi:10.1080/00223890802634266
   PubMed Web of Science ®Google Scholar
• Schweizer, K. (2012). On correlated errors. European Journal of Psychological Assessment, 28, 1–2. doi:10.1027/1015-5759/a000094
   Web of Science ®Google Scholar
• Sliter, K. A., & Zickar, M. J. (2014). An IRT examination of the psychometric functioning of negatively worded personality items. Educational and Psychological Measurement, 74, 214–226. doi:10.1177/0013164413504584
   Web of Science ®Google Scholar
• Stucky, B. D., & Edelen, M. O. (2014). Using hierarchical IRT models to create unidimensional measures from multidimensional data. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183–206). New York, NY: Routledge/Taylor & Francis Group.
   Google Scholar
• Tomás, J. M., & Oliver, A. (1999). Rosenberg’s Self-Esteem Scale: Two factors or method effects. Structural Equation Modeling, 6, 84–98. doi:10.1080/10705519909540120
   Web of Science ®Google Scholar
• Tomás, J. M., Oliver, A., Galiana, L., Sancho, P., & Lila, M. (2013). Explaining method effects associated with negatively worded items in trait and state global and domain-specific self-esteem scales. Structural Equation Modeling, 20, 299–313. doi:10.1080/10705511.2013.769394
   Web of Science ®Google Scholar
• Vecchione, M., Alessandri, G., Caprara, G. V., & Tisak, J. (2014). Are method effects permanent or ephemeral in nature? The case of the revised life orientation test. Structural Equation Modeling, 21, 117–130. doi:10.1080/10705511.2014.859511
   Web of Science ®Google Scholar
• Wang, Y., Kong, F., Huang, L., & Liu, J. (2016). Neural correlates of biased responses: The negative method effect in the Rosenberg Self-Esteem Scale is associated with right amygdala volume. Journal of Personality, 84, 623–632. doi:10.1111/jopy.
   PubMed Web of Science ®Google Scholar
• Weijters, B., Baumgartner, H., & Schillewaert, N. (2013). Reversed item bias: An integrative model. Psychological Methods, 18, 320–334. doi:10.1037/a0032121
   PubMed Web of Science ®Google Scholar
• Wu, Y., Wen, Z., Marsh, H. W., & Hau, K.-T. (2013). A comparison of strategies for forming product indicators for unequal numbers of items in structural equation models of latent interactions. Structural Equation Modeling, 20, 551–567. doi:10.1080/10705511.2013.824772
   Web of Science ®Google Scholar
• Xin, Z., & Chi, L. (2010). Wording effect leads to a controversy over the construct of the social dominance orientation scale. The Journal of Psychology, 144, 473–488. doi:10.1080/00223980.2010.496672
   PubMed Web of Science ®Google Scholar
• Ye, S. (2009). Factor structure of the General Health Questionnaire (GHQ–12): The role of wording effects. Personality and Individual Differences, 46, 197–201. doi:10.1016/j.paid.2008.09.027
   Web of Science ®Google Scholar