Examining the Wording Effect: What Are We Measuring?

References

• Adkins-Wood, D. (1961). Test construction. Merrill.
   Google Scholar
• Anastasi, A. (1980). Psychological testing. MacMillan.
   Google Scholar
• Baumgartner, H., & Steenkamp, J.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143 156. https://doi.org/10.1509/jmkr.38.2.143.18840
   Web of Science ®Google Scholar
• Benson, J., & Hocevar, D. (1985). The impact of item phrasing on the validity of attitude scales for elementary school children. Journal of Educational Measurement, 22(3), 231–240. https://doi.org/10.1111/j.1745-3984.1985.tb01061.x
   Web of Science ®Google Scholar
• Bentler, P. M., Jackson, D. N., & Messick, S. (1971). Identification of content and style: A two-dimensional interpretation of acquiescence. Psychological Bulletin, 76(3), 186–204. https://doi.org/10.1037/h0031474
   PubMedGoogle Scholar
• Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford
   Google Scholar
• Bulut, H. C. (2021). Item wording effects in psychological measures: Do early literacy skills matter? Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 12(3), 239–253. https://doi.org/10.21031/epod.944067
   Google Scholar
• Carlson, D. S., Ferguson, M., Kacmar, K. M., Grzywacz, J. G., & Whitten, D. (2011). Pay it forward: The positive crossover effects of supervisor work—family enrichment. Journal of Management, 37(3), 770–789. https://doi.org/10.1177/0149206310363613
   Web of Science ®Google Scholar
• Çetin, B., Doğan, T., & Sapmaz, F. (2010). Olumsuz değerlendirilme korkusu ölçeği kısa formunun Türkçe uyarlaması: Geçerlik ve güvenirlik çalışması. Eğitim ve Bilim, 35(156), 205–216.
   Google Scholar
• Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
   Web of Science ®Google Scholar
• Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5
   Web of Science ®Google Scholar
• Cho, E. (2022). Reliacoef: Compute and compare unidimensional and multidimensional reliability coefficients. https://rdrr.io/github/eunscho/reliacoef/
   Google Scholar
• DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage.
   Google Scholar
• Diedenhofen, B., & Musch, J. (2016). Cocron: A web interface and R package for the statistical comparison of cronbach’s alpha coefficients. International Journal of Internet Science, 11(1), 51–60.
   Web of Science ®Google Scholar
• Dodeen, H. (2015). The effects of positively and negatively worded items on the factor structure of the UCLA loneliness scale. Journal of Psychoeducational Assessment, 33(3), 259–267. https://doi.org/10.1177/0734282914548325
   Web of Science ®Google Scholar
• Essau, C. A., Olaya, B., Anastassiou‐Hadjicharalambous, X., Pauli, G., Gilvarry, C., Bray, D., O’callaghan, J., & Ollendick, T. H. (2012). Psychometric properties of the strength and difficulties questionnaire from five European countries. International Journal of Methods in Psychiatric Research, 21(3), 232–245. https://doi.org/10.1002/mpr.1364
   PubMed Web of Science ®Google Scholar
• Field, A. (2009). Discovering statistics using SPSS (3rd ed.). Sage.
   Google Scholar
• García-Fernández, J., Postigo, Á., Cuesta, M., González-Nuevo, C., Menéndez-Aller, Á., & García-Cueto, E. (2022). To be direct or not: Reversing likert response format items. The Spanish Journal of Psychology, 25, e24. https://doi.org/10.1017/SJP.2022.20
   PubMed Web of Science ®Google Scholar
• Gliner, J. A., Morgan, G. A., & Leech, N. L. (2017). Research methods in applied settings: An integrated approach to design and analysis. Routledge.
   Google Scholar
• Gölbaşı Şimşek, G., & Noyan, F. (2013). McDonald’s ømega t, Cronbach’s α, and generalized þeta for composite reliability of common factors structures. Communications in Statistics - Simulation & Computation, 42(9), 2008–2025. https://doi.org/10.1080/03610918.2012.689062
   Web of Science ®Google Scholar
• Greenberger, E., Chen, C., Dmitrieva, J., & Farruggia, S. P. (2003). Item-wording and the dimensionality of the Rosenberg self-esteem scale: Do they matter? Personality and Individual Differences, 35(6), 1241–1254. https://doi.org/10.1016/S0191-8869(02)00331-8
   Web of Science ®Google Scholar
• Gu, H., Wen, Z., & Fan, X. (2015). The impact of wording effect on reliability and validity of the Core Self-Evaluation Scale (CSES): A bi-factor perspective. Personality and Individual Differences, 83, 142–147. https://doi.org/10.1016/j.paid.2015.04.006
   Web of Science ®Google Scholar
• Harpole, J. K., Levinson, C. A., Woods, C. M., Rodebaugh, T. L., Weeks, J. W., Brown, P. J., Heimberg, R. G., Menatti, A. R., Blanco, C., Schneier, F., & Liebowitz, M. (2015). Assessing the straightforwardly-worded brief fear of negative evaluation scale for differential item functioning across gender and ethnicity. Journal of Psychopathology and Behavioral Assessment, 37(2), 306–317. https://doi.org/10.1007/s10862-014-9455-9
   PubMed Web of Science ®Google Scholar
• Hughes, G. D. (2009). The impact of incorrect responses to reverse-coded survey items. Research in the Schools, 16(2), 76–88.
   Google Scholar
• Kail, R., & Park, Y. S. (1990). Impact of practice on speed of mental rotation. Journal of Experimental Child Psychology, 49(2), 227–244. https://doi.org/10.1016/0022-0965(90)90056-E
   PubMed Web of Science ®Google Scholar
• Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31–36. https://doi.org/10.1007/BF02291575
   Web of Science ®Google Scholar
• Kılıç, A. F. (2022). Ölçek geliştirme sürecinde açımlayıcı faktör analizi. In M. Acar-Güvendir & Y. Özer-Özkan (Eds.), Tüm yönleriyle ölçek geliştirme süreci (pp. 69–129). Pegem Akademi.
   Google Scholar
• Kılıç, A. F., & Doğan, N. (2021). Comparison of confirmatory factor analysis estimation methods on mixed-format data. International Journal of Assessment Tools in Education, 8(1), 21–37. https://doi.org/10.21449/ijate.782351
   Web of Science ®Google Scholar
• Leary, M. R. (1983). A brief version of the fear of negative evaluation scale. Personality and Social Psychology Bulletin, 9(3), 371–375. https://doi.org/10.1177/0146167283093007
   Web of Science ®Google Scholar
• Li, H., & Liu, H. (2021). Beginning EFL teachers’ emotional labor strategies in the Chinese context. Frontiers in Psychology, 12, 737746. https://doi.org/10.3389/fpsyg.2021.737746
   PubMed Web of Science ®Google Scholar
• Lindwall, M., Barkoukis, V., Grano, C., Lucidi, F., Raudsepp, L., Liukkonen, J., & Thøgersen-Ntoumani, C. (2012). Method effects: The problem with negatively versus positively keyed items. Journal of Personality Assessment, 94(2), 196–204. https://doi.org/10.1080/00223891.2011.645936
   PubMed Web of Science ®Google Scholar
• Marin, G., Gamba, R. J., & Marin, B. V. (1992). Extreme response style and acquiescence among hispanics: The role of acculturation and education. Journal of Cross-Cultural Psychology, 23(4), 498–509. https://doi.org/10.1177/0022022192234006
   Web of Science ®Google Scholar
• Marsh, H. W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon. Developmental Psychology, 22(1), 37–49. https://doi.org/10.1037/0012-1649.22.1.37
   Web of Science ®Google Scholar
• Menold, N. (2020). How do reverse-keyed items in inventories affect measurement quality and information processing? Field Methods, 32(2), 140–158. https://doi.org/10.1177/1525822X19890827
   Web of Science ®Google Scholar
• Merritt, S. M. (2012). The two-factor solution to Allen and Meyer’s (1990) affective commitment scale: Effects of negatively worded items. Journal of Business and Psychology, 27(4), 421–436. https://doi.org/10.1007/s10869-011-9252-3
   Web of Science ®Google Scholar
• Meule, A., Reichenberger, J., & Blechert, J. (2018). Development and preliminary validation of the Salzburg emotional eating scale. Frontiers in Psychology, 9, 88. https://doi.org/10.3389/fpsyg.2018.00088
   PubMed Web of Science ®Google Scholar
• Mirowsky, J., & Ross, C. E. (1991). Eliminating defense and agreement bias from measures of the sense of control: A 2 x 2 index. Social Psychology Quarterly, 54(2), 127–145. https://doi.org/10.2307/2786931
   Web of Science ®Google Scholar
• Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGrawHill.
   Google Scholar
• Osborne, J. W. (2014). Best practices in exploratory factor analysis. CreateSpace Independent Publishing Platform. https://doi.org/10.4135/9781412995627.d8
   Google Scholar
• Parent, J., & Forehand, R. (2017). The multidimensional assessment of parenting scale (MAPS): Development and psychometric properties. Journal of Child and Family Studies, 26(8), 2136–2151. https://doi.org/10.1007/s10826-017-0741-5
   PubMed Web of Science ®Google Scholar
• Paulhus, D. L., & Reid, D. B. (1991). Enhancement and denial in socially desirable responding. Journal of Personality and Social Psychology, 60(2), 307–317. https://doi.org/10.1037/0022-3514.60.2.307
   Web of Science ®Google Scholar
• Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879
   PubMed Web of Science ®Google Scholar
• Prentice, D. A. (1987). Psychological correspondence of possessions, attitudes, and values. Journal of Personality and Social Psychology, 53(6), 993–1003. https://doi.org/10.1037/0022-3514.53.6.993
   Web of Science ®Google Scholar
• Revelle, W. (2022). Psych: Procedures for psychological, psychometric, and personality research (2.2.9) [Computer software]. Northwestern University. https://cran.r-project.org/package=psych
   Google Scholar
• Rosenthal, L. J., Fulton, B. J., Hirsch, L. A., Isaacson, H. T., Howard, A. W., Dedrick, C. M., Sherstyuk, I. A., Blunt, S. C., Petigura, E. A., Knutson, H. A., Behmard, A., Chontos, A., Crepp, J. R., Crossfield, I. J. M., Dalba, P. A., Fischer, D. A., Henry, G. W., Kane, S. R., Kosiarek, M. … Wright, J. T. (2021). The California legacy survey. I. A catalog of 178 planets from precision radial velocity monitoring of 719 nearby stars over three decades. The Astrophysical Journal Supplement Series, 255(1), 8–67. https://doi.org/10.3847/1538-4365/abe23c
   Web of Science ®Google Scholar
• Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
   Web of Science ®Google Scholar
• Roszkowski, M. J., & Soven, M. (2010). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35(1), 113–130. https://doi.org/10.1080/02602930802618344
   Web of Science ®Google Scholar
• Salazar, M. S. (2015). The dilemma of combining positive and negative items in scales. Psicothema, 27(2), 192–199. https://doi.org/10.7334/psicothema2014.266
   PubMed Web of Science ®Google Scholar
• Sauro, J., & Lewis, J. R. (2011). When designing usability questionnaires, does it hurt to be positive? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2215–2224). https://doi.org/10.1145/1978942.1979266
   Google Scholar
• Schei, G. S., Haugen, T., Stenling, A., Grøtting, A., Peters, D. M., & Høigaard, R. (2021). Development and initial validation of the humor climate in sport scale. Frontiers in Psychology, 12, 692892. https://doi.org/10.3389/fpsyg.2021.692892
   PubMedGoogle Scholar
• Schriesheim, C. A., & Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41(4), 1101–1114. https://doi.org/10.1177/001316448104100420
   Web of Science ®Google Scholar
• Schütze, R., Rees, C., Smith, A., Slater, H., Catley, M., & O’Sullivan, P. (2019). Assessing beliefs underlying rumination about pain: Development and validation of the pain metacognitions questionnaire. Frontiers in Psychology, 10, 910. https://doi.org/10.3389/fpsyg.2019.00910
   PubMed Web of Science ®Google Scholar
• Stewart, T. J., & Frye, A. W. (2004). Investigating the use of negatively phrased survey items in medical education settings: Common wisdom or common mistake? Academic Medicine, 79(10), S18–S20. https://doi.org/10.1097/00001888-200410001-00006
   PubMed Web of Science ®Google Scholar
• Suárez-Álvarez, J., Pedrosa, I., Lozano, L. M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in Likert scales: A questionable practice. Psicothema, 30(2), 149–158. https://doi.org/10.7334/psicothema2018.33
   PubMed Web of Science ®Google Scholar
• Thorndike, R. L. (1971). Educational measurement. American Council on Education.
   Google Scholar
• Uysal, İ., & Kılıç, A. F. (2022a). Çok değişkenli normallik testleri üzerine bir inceleme. In EjerCongress (pp. 968–969).
   Google Scholar
• Uysal, İ., & Kılıç, A. F. (2022b). Normal dağılım ikilemi. Anadolu Journal of Educational Sciences International, 12(1), 220–248. https://doi.org/10.18039/ajesi.962653
   Google Scholar
• Van Sonderen, E., Sanderman, R., & Coyne, J. C. (2013). Correction: Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. Public Library of Science One, 8(9), e68967. https://doi.org/10.1371/journal.pone.0068967
   Google Scholar
• Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4–70. https://doi.org/10.1177/109442810031002
   Web of Science ®Google Scholar
• Venta, A., Bailey, C. A., Walker, J., Mercado, A., Colunga-Rodriguez, C., Ángel-González, M., & Dávalos-Picazo, G. (2022). Reverse-coded items do not work in Spanish: Data from four samples using established measures. Frontiers in Psychology, 13, 828037. https://doi.org/10.3389/fpsyg.2022.828037
   PubMed Web of Science ®Google Scholar
• Vigil-Colet, A., Navarro-González, D., & Morales-Vives, F. (2020). To reverse or to not reverse likert-type items: That is the question. Psicothema, 32(1), 108–114. https://doi.org/10.7334/psicothema2019.286
   PubMed Web of Science ®Google Scholar
• Watson, D. (1992). Correcting for acquiescent response bias in the absence of a balanced scale: An application to class consciousness. Sociological Methods & Research, 21(1), 52–88. https://doi.org/10.1177/0049124192021001003
   Web of Science ®Google Scholar
• Weems, G. H., Onwuegbuzie, A. J., & Lustig, D. (2003). Profiles of respondents who respond inconsistently to positively-and negatively-worded items on rating scales. Evaluation & Research in Education, 17(1), 45–60. https://doi.org/10.1080/14664200308668290
   Google Scholar
• Weijters, B., & Baumgartner, H. (2012). Misresponse to reversed and negated items in surveys: A review. Journal of Marketing Research, 49(5), 737–747. https://doi.org/10.1509/jmr.11.0368
   Web of Science ®Google Scholar
• Weijters, B., Baumgartner, H., & Schillewaert, N. (2013). Reversed item bias: An integrative model. Psychological Methods, 18(3), 320–334. https://doi.org/10.1037/a0032121
   PubMed Web of Science ®Google Scholar
• Wiggins, J. S. (1973). Personality and prediction. Addison-Wesley.
   Google Scholar
• Wong, N., Rindfleisch, A., & Burroughs, J. E. (2003). Do reverse-worded items confound measures in cross-cultural consumer research? The case of the material values scale. Journal of Consumer Research, 30(1), 72–91. https://doi.org/10.1086/374697
   Web of Science ®Google Scholar
• Zhang, X., Noor, R., Savalei, V., & Windmann, S. (2016). Examining the effect of reverse worded items on the factor structure of the need for cognition scale. Public Library of Science One, 11(6), e0157795. https://doi.org/10.1371/journal.pone.0157795
   Google Scholar