The Development and Validation of a Multidimensional Forced-Choice Format Character Measure: Testing the Thurstonian IRT Approach

References

• Allport, G. (1937). Personality: A psychological interpretation. Oxford, England: Holt.
   Google Scholar
• Brown, A., Inceoglu, I., & Lin, Y. (2017). Preventing rater biases in 360-Degree feedback by forcing choice. Organizational Research Methods, 20(1), 121–148. doi:10.1177/1094428116668036
   Web of Science ®Google Scholar
• Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502. doi:10.1177/0013164410375112
   Web of Science ®Google Scholar
• Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36–52. doi:10.1037/a0030641
   PubMed Web of Science ®Google Scholar
• Bürkner, P., Schulte, N., & Holling, H. (2019). On the statistical and practical limitations of Thurstonian IRT models. Educational and Psychological Measurement, 79(5), 827–854. doi:10.1177/0013164419832063
   PubMed Web of Science ®Google Scholar
• Campbell, D. T., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 59(2), 81–105. doi:10.1037/h0046016
   Web of Science ®Google Scholar
• Campbell, D. T. (1960). Recommendations for APA test standards regarding construct, trait, or discriminant validity. American Psychologist, 15(8), 546–553. doi:10.1037/h0048255
   Web of Science ®Google Scholar
• Cao, M., & Drasgow, F. (2019, May 9). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347–1368. doi:10.1037/apl0000414
   PubMed Web of Science ®Google Scholar
• Chernyshenko, O. S., Stark, S., Prewett, M. S., Gray, A. A., Stilson, F. R., & Tuttle, M. D. (2009). Normative scoring of multidimensional pairwise preference personality scales using IRT: Empirical comparisons with other formats. Human Performance, 22(2), 105–127. doi:10.1080/08959280902743303
   Web of Science ®Google Scholar
• Cheung, M. W.-L., & Chan, W. (2002). Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 9(1), 55–77. doi:10.1207/S15328007SEM0901_4
   Web of Science ®Google Scholar
• Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267–307. doi:10.1207/s15327043hup1803_4.
   Web of Science ®Google Scholar
• Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.
   PubMed Web of Science ®Google Scholar
• Cohen, T. R., Panter, A. T., Turan, N., Morse, L., & Kim, Y. (2014). Moral character in the workplace. Journal of Personality and Social Psychology, 107(5), 943–963.
   PubMed Web of Science ®Google Scholar
• Converse, P. D., Oswald, F. L., Imus, A., Hedricks, C., Roy, R., & Butera, H. (2008). Comparing personality test formats and warnings: Effects on criterion-related validity and test-taker reactions. International Journal of Selection and Assessment, 16(2), 155–169. doi:10.1111/j.1468-2389.2008.00420.x.
   Web of Science ®Google Scholar
• Dahlsgaard, K., Peterson, C., & Seligman, M. E. P. (2005). Shared virtue: The convergence of valued human strengths across culture and history. Review of General Psychology, 9(3), 203–213. doi:10.1037/1089-2680.9.3.203.
   Web of Science ®Google Scholar
• Davis, M. H. (1983). Measuring individual differences in empathy: Evidence for a multidimensional approach. Journal of Personality and Social Psychology, 44(1), 113–126. doi:10.1037/0022-3514.44.1.113.
   Web of Science ®Google Scholar
• Doris, J. M. (1998). Persons, situations, and virtue ethics. Nous, 34(2), 504–530.
   Web of Science ®Google Scholar
• Drasgow, F., & Kang, T. (1984). Statistical power of differential validity and differential prediction analyses for detecting measurement nonequivalence. Journal of Applied Psychology, 69(3), 498–508. doi:10.1037/0021-9010.69.3.498
   Web of Science ®Google Scholar
• Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087–1101. doi:10.1037/0022-3514.92.6.1087
   PubMed Web of Science ®Google Scholar
• Dueber, D. M., Love, A. M. A., Toland, M. D., & Turner, T. A. (2019). Comparison of single-response format and forced-choice format instruments using Thurstonian item response theory. Educational and Psychological Measurement, 79(1), 108–128. doi:10.1177/0013164417752782
   PubMed Web of Science ®Google Scholar
• Dunnette, M. D., Mccartney, J., Carlson, H. C., & Kirchner, W. K. (1962). A study of faking behavior on a forced-choice self-description checklist. Personnel Psychology, 15(1), 13–24. doi:10.1111/j.1744-6570.1962.tb01843.x
   Web of Science ®Google Scholar
• Fisher, P., Robie, C., Christiansen, N., Speer, A., & Schneider, L. (2019). Criterion-related validity of forced-choice personality measures: A cautionary note regarding Thurstonian IRT versus classical test theory scoring. Personnel Assessment and Decisions, 5(1), 49–61. doi:10.25035/pad.2019.01.003
   Google Scholar
• Fleeson, W. (2001). Toward a structure- and process-integrated view of personality: Traits as density distributions of states. Journal of Personality and Social Psychology, 80(6), 1011–1027. doi:10.1037//0022-3514.80.6.1011
   PubMed Web of Science ®Google Scholar
• Fleeson, W., Furr, R. M., Jayawickreme, E., Meindl, P., & Helzer, E. G. (2014). Character: The prospects for a personality-based perspective on morality. Social and Personality Psychology Compass, 8(4), 178–191. doi:10.1111/spc3.12094
   Google Scholar
• Fowers, B. J. (2014). Toward programmatic research on virtue assessment: Challenges and prospects. Theory and Research in Education, 12(3), 309–328. doi:10.1177/1477878514546064
   Google Scholar
• Goffin, R. D., & Boyd, A. C. (2009). Faking and personality assessment in personnel selection: Advancing models of faking. Canadian Psychology/Psychologie Canadienne, 50(3), 151–160. doi:10.1037/a0015946
   Web of Science ®Google Scholar
• Guenole, N., Brown, A. A., & Cooper, A. J. (2018). Forced-choice assessment of work-related maladaptive personality traits: Preliminary evidence from an application of Thurstonian item response modeling. Assessment, 25(4), 513–526. doi:10.1177/1073191116641181
   PubMed Web of Science ®Google Scholar
• Harzer, C., & Ruch, W. (2014). The role of character strengths for task performance, job dedication, interpersonal facilitation, and organizational support. Human Performance, 27(3), 183–205. doi:10.1080/08959285.2014.913592
   Web of Science ®Google Scholar
• Heggestad, E. D., Morrison, M., Reeve, C. L., & Mccloy, R. A. (2006). Forced-choice assessments of personality for selection: Evaluating issues of normative assessment and faking resistance. Journal of Applied Psychology, 91(1), 9–24. doi:10.1037/0021-9010.91.1.9
   PubMed Web of Science ®Google Scholar
• Helzer, E. G., Furr, R. M., Hawkins, A., Barranti, M., Blackie, L. E. R., & Fleeson, W. (2014). Agreement on the perception of moral character. Personality and Social Psychology Bulletin, 40(12), 1698–1710. doi:10.1177/0146167214554957
   PubMed Web of Science ®Google Scholar
• Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167–184. doi:10.1037/h0029780
   Web of Science ®Google Scholar
• Hogan, J., Barrett, P., & Hogan, R. (2007). Personality measurement, faking, and employment selection. Journal of Applied Psychology, 92(5), 1270–1285. doi:10.1037/0021-9010.92.5.1270
   PubMed Web of Science ®Google Scholar
• Hontangas, P. M., de la Torre, J., Ponsoda, V., Leenen, I., Morillo, D., & Abad, F. J. (2015). Comparing traditional and IRT scoring of forced-choice Tests. Applied Psychological Measurement, 39(8), 598–612. doi:10.1177/0146621615585851
   PubMed Web of Science ®Google Scholar
• Johnson, C. E., Wood, R., & Blinkhorn, S. F. (1988). Spuriouser and spuriouser: The use of ipsative personality tests. Journal of Occupational Psychology, 61(2), 153–162. doi:10.1111/j.2044-8325.1988.tb00279.x
   Web of Science ®Google Scholar
• Kashdan, T. B., Gallagher, M. W., Silvia, P. J., Winterstein, B. P., Breen, W. E., Terhar, D., & Steger, M. F. (2009). The Curiosity and Exploration Inventory-II: Development, factor structure, and psychometrics. Journal of Research in Personality, 43(6), 987–998. doi:10.1016/j.jrp.2009.04.011
   PubMed Web of Science ®Google Scholar
• Lee, P., Lee, S., & Stark, S. (2018). Examining validity evidence for multidimensional forced choice measures with different scoring approaches. Personality and Individual Differences, 123, 229–235. doi:10.1016/j.paid.2017.11.031
   Web of Science ®Google Scholar
• Leonard, H. S. (1997). The many faces of character. Consulting Psychology Journal: Practice and Research, 49(4), 235–245. doi:10.1037//1061-4087.49.4.235
   Google Scholar
• Martin, B. A., Bowen, C.-C., & Hunt, S. T. (2002). How effective are people at faking on personality questionnaires? Personality and Individual Differences, 32(2), 247–256. doi:10.1016/S0191-8869(01)00021-6
   Web of Science ®Google Scholar
• McCloy, R. A., Heggestad, E. D., & Reeve, C. L. (2005). A silk purse from the sow’s ear: Retrieving normative information from multidimensional forced-choice items. Organizational Research Methods, 8(2), 222–248. doi:10.1177/1094428105275374
   Web of Science ®Google Scholar
• McCullough, M. E., Emmons, R. A., & Tsang, J.-A. (2002). The grateful disposition: A conceptual and empirical topography. Journal of Personality and Social Psychology, 82(1), 112–127. doi:10.1037//0022-3514.82.1.112
   PubMed Web of Science ®Google Scholar
• McCullough, M. E., & Snyder, C. R. (2000). Classical sources of human strength: Revisiting an old home and building a new one. Journal of Social and Clinical Psychology, 19(1), 1–10. doi:10.1521/jscp.2000.19.1.1
   Web of Science ®Google Scholar
• McFarland, L. A., & Ryan, A. M. (2000). Variance in faking across noncognitive measures. Journal of Applied Psychology, 85(5), 812–821. doi:10.1037/0021-9010.85.5.812
   PubMed Web of Science ®Google Scholar
• McFarland, L. A., & Ryan, A. M. (2006). Toward an integrated model of applicant faking behavior. Journal of Applied Social Psychology, 36(4), 979–1016. doi:10.1111/j.0021-9029.2006.00052.x
   Web of Science ®Google Scholar
• McGrath, R. E., Hall-Simmonds, A., & Goldberg, L. R. (2020). Are measures of character and personality distinct? Evidence from observed-score and true-score analyses. Assessment, 27, 117–135. doi:10.1177/1073191117738047
   PubMed Web of Science ®Google Scholar
• Meade, A. W. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77(4), 531–551. doi:10.1348/0963179042596504
   Web of Science ®Google Scholar
• Meindl, P., Jayawickreme, E., Furr, R. M., & Fleeson, W. (2015). A foundation beam for studying morality from a personological point of view: Are individual differences in moral behaviors and thoughts consistent? Journal of Research in Personality, 59, 81–92. doi:10.1016/j.jrp.2015.09.005
   Web of Science ®Google Scholar
• Mischel, W. (1968). Personality and assessment. New York, NY: Wiley.
   Google Scholar
• Morillo, D., Leenen, I., Abad, F. J., Hontangas, P., de la Torre, J., & Ponsoda, V. (2016). A dominance variant under the multi-unidimensional pairwise-preference framework: Model formulation and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 40(7), 500–516. doi:10.1177/0146621616662226
   PubMed Web of Science ®Google Scholar
• Mueller-Hanson, R., Heggestad, E. D., & Thornton, G. C. I. (2003). Faking and selection: Considering the use of personality from select-in and select-out perspectives. Journal of Applied Psychology, 88(2), 348–355. doi:10.1037/0021-9010.88.2.348
   PubMed Web of Science ®Google Scholar
• Mullins-Sweatt, S. N., Jamerson, J. E., Samuel, D. B., Olson, D. R., & Widiger, T. A. (2006). Psychometric properties of an abbreviated instrument of the five-factor model. Assessment, 13(2), 119–137. doi:10.1177/1073191106286748
   PubMed Web of Science ®Google Scholar
• Muthén, L. K., & Muthén, B. O. (1998–2014). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén. Retrieved from www.statmodel.com
   Google Scholar
• Nederhof, J. A. (1985). Methods of coping with social desirability: A review. European Journal of Social Psychology, 15(3), 263–280. doi:10.1002/ejsp.2420150303
   Web of Science ®Google Scholar
• Ng, V., Tay, L., & Kuykendall, L. (2018). The development and validation of a measure of character: The CIVIC. The Journal of Positive Psychology, 13(4), 346–372. doi:10.1080/17439760.2017.1291850
   Web of Science ®Google Scholar
• Niemic, R. M. (2013). VIA character strengths: Research and practice (The first 10 years). In H. H. Knoop & A. D. Fave (Eds.), Well-being and cultures: Cross-cultural advancements in positive psychology (Vol. 3, pp. 11–29). Dordrecht, NL: Springer.
   Google Scholar
• O’Neill, T. A., Lewis, R. J., Law, S. J., Larson, N., Hancock, S., Radan, J., … Carswell, J. J. (2017). Forced-choice pre-employment personality assessment: Construct validity and resistance to faking. Personality and Individual Differences, 115, 120–127. doi:10.1016/j.paid.2016.03.075
   Web of Science ®Google Scholar
• Park, N., Peterson, C., & Seligman, M. (2004). Strengths of character and well-being. Journal of Social and Clinical Psychology, 23(5), 603–619. doi:10.1521/jscp.23.5.603.50748
   Web of Science ®Google Scholar
• Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46(3), 598–609. doi:10.1037/0022-3514.46.3.598
   Web of Science ®Google Scholar
• Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (Vol. 1, pp. 17–59). San Diego, CA: Academic Press.
   Google Scholar
• Pavlov, G., Maydeu-Olivares, A., & Fairchild, A. J. (2019). Effects of applicant faking on forced-choice and Likert scores. Organizational Research Methods, 22(3), 710–739. doi:10.1177/1094428117753683
   Web of Science ®Google Scholar
• Peterson, C. E., & Seligman, M. E. P. (2004). Character strengths and virtues. New York, NY: Oxford University Press.
   Google Scholar
• Piedmont, R. L. (1999). Does spirituality represent the sixth factor of personality? Spiritual transcendence and the Five-Factor Model. Journal of Personality, 67(6), 985–1013. doi:10.1111/1467-6494.00080
   Web of Science ®Google Scholar
• Robertson, D. (2013). Stoicism and the art of happiness. London: Teach Yourself.
   Google Scholar
• Robinson, S. L., & Bennett, R. J. (1995). A typology of deviant workplace behaviors: A multidimensional scaling study. Academy of Management Journal, 38(2), 555–572. doi:10.2307/256693
   Web of Science ®Google Scholar
• Rosse, J. G., Stecher, M. D., Miller, J. L., & Levin, R. A. (1998). The impact of response distortion on preemployment personality testing and hiring decisions. Journal of Applied Psychology, 83(4), 634–644. doi:10.1037/0021-9010.83.4.634
   Web of Science ®Google Scholar
• Rothstein, M. G., & Goffin, R. D. (2006). The use of personality measures in personnel selection: What does current research support? Human Resource Management Review, 16(2), 155–180. doi:10.1016/j.hrmr.2006.03.004
   Google Scholar
• Ruch, W., Proyer, R. T., Harzer, C., Park, N., Peterson, C., & Seligman, M. E. P. (2010). Values in Action Inventory of Strengths (VIA-IS): Adaptation and validation of the German version and the development of a peer-rating form. Journal of Individual Differences, 31(3), 138–149. doi:10.1027/1614-0001/a000022
   Web of Science ®Google Scholar
• Salgado, J. F., Anderson, N. R., & Táuriz, G. (2014). The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journal of Occupational and Organizational Psychology, 88(4), 797–834. doi:10.1111/joop.12098
   Web of Science ®Google Scholar
• Salgado, J. F., & Táuriz, G. (2014). The Five-Factor Model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23(1), 3–30. doi:10.1080/1359432X.2012.716198
   Web of Science ®Google Scholar
• Sarason, B. R., Sarason, I. G., Hacker, T. A., & Basham, R. B. (1985). Concomitants of social support: Social skills, physical attractiveness, and gender. Journal of Personality and Social Psychology, 49(2), 469–480. doi:10.1037/0022-3514.49.2.469
   Web of Science ®Google Scholar
• Schmit, M. J., & Ryan, A. M. (1993). The Big Five in personnel selection-factor structure in applicant and nonapplicant populations. Journal of Applied Psychology, 78(6), 966–974. doi:10.1037/0021-9010.78.6.966
   Web of Science ®Google Scholar
• Smither, J. W., Reilly, R. R., Millsap, R. E., Pearlman, K., & Stoffey, R. W. (1993). Applicant reactions to selection procedures. Personnel Psychology, 46(1), 49–76. doi:10.1111/j.1744-6570.1993.tb00867.x
   Web of Science ®Google Scholar
• Snell, A. F., Sydell, E. J., & Lueke, S. B. (1999). Towards a theory of applicant faking: Integrating studies of deception. Human Resource Management Review, 9(2), 219–242. doi:10.1016/S1053-4822(99)00019-4
   Web of Science ®Google Scholar
• Sollman, M. J., Ranseen, J. D., & Berry, D. T. R. (2010). Detection of feigned ADHD in college students. Psychological Assessment, 22(2), 325–335. doi:10.1037/a0018857
   PubMed Web of Science ®Google Scholar
• Stanush, P. L. (1997). Factors that influence the susceptibility of self-report inventories to distortion: A meta-analytic investigation (Unpublished (doctoral dissertation). College Stations, TX Texas A&M University
   Google Scholar
• Stark, S., Chernyshenko, O. S., Chan, K.-Y., Lee, W. C., & Drasgow, F. (2001). Effects of the testing situation on item responding: Cause for concern. Journal of Applied Psychology, 86(5), 943–953. doi:10.1037/0021-9010.86.5.943
   PubMed Web of Science ®Google Scholar
• Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. doi:10.1177/0146621604273988
   Web of Science ®Google Scholar
• Stark, S., Chernyshenko, O. S., & Drasgow, F. (2012). Constructing fake-resistant personality tests using item response theory: High stakes personality testing with multidimensional pairwise preferences. In M. Ziegler, C. MacCann, & R. D. Roberts (Eds.), New perspectives on faking in personality assessments (pp. 215–239). New York, NY: Oxford University Press.
   Google Scholar
• Stark, S., Chernyshenko, O. S., Drasgow, F., & Williams, B. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring? Journal of Applied Psychology, 91(1), 25–39. doi:10.1037/0021-9010.91.1.25
   PubMed Web of Science ®Google Scholar
• Strathman, A., Gleicher, F., Boninger, D. S., & Edwards, C. S. (1994). The consideration of future consequences: Weighing immediate and distant outcomes of behavior. Journal of Personality and Social Psychology, 66(4), 742–752. doi:10.1037//0022-3514.66.4.742
   Web of Science ®Google Scholar
• Su, R., Tay, L., & Diener, E. (2014). The development and validation of the Comprehensive Inventory of Thriving (CIT) and the Brief Inventory of Thriving (BIT). Applied Psychology: Health and Well-Being, 6(3), 251–279. doi:10.1111/aphw.12027
   PubMed Web of Science ®Google Scholar
• Thompson, A. D., & Riggio, R. E. (2010). Introduction to special issue on defining and measuring character. Consulting Psychology Journal: Practice and Research, 62(4), 211–215. doi:10.1037/a0022285
   Google Scholar
• Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286. doi:10.1037/h0070288
   Google Scholar
• Thurstone, L. L. (1931). Rank order as a psychophysical method. Journal of Experimental Psychology, 14(3), 187–201. doi:10.1037/h0070025
   Google Scholar
• Walton, K. E., Cherkasova, L., & Roberts, R. D. (2019). On the validity of forced choice scores derived from the Thurstonian item response theory model. Assessment. doi:10.1177/1073191119843585
   PubMed Web of Science ®Google Scholar
• Wang, W. C., Qiu, X. L., Chen, C. W., Ro, S., & Jin, K. Y. (2017). Item response theory models for ipsative tests with multidimensional pairwise comparison items. Applied Psychological Measurement, 41(8), 600–613. doi:10.1177/0146621617703183
   PubMed Web of Science ®Google Scholar
• Waters, L. K. (1965). A note on the “fakability” of forced‐choice scales. Personnel Psychology, 18(2), 187–191. doi:10.1111/j.1744-6570.1965.tb00277.x
   Web of Science ®Google Scholar
• Wetzel, E., & Frick, S. (2020). Comparing the validity of trait estimates from the multidimensional forced-choice format and the rating scale format. Psychological Assessment, 32(3), 239–253. doi:10.1037/pas0000781
   PubMed Web of Science ®Google Scholar
• Wiegand, J. P. (2015). Using Amazon’s MTURK for Multiple Waves of Data Collection: Part 2 (Qualifications, Follow-up Survey Invitations and Tracking Responses between Surveys) [Blog post]. Retrieved from http://justinwiegand.com/blog/?p=211.
   Google Scholar
• Wood, A. M., Linley, P. A., Maltby, J., Baliousis, M., & Joseph, S. (2008). The authentic personality: A theoretical and empirical conceptualization and the development of the Authenticity Scale. Journal of Counseling Psychology, 55(3), 385–399. doi:10.1037/0022-0167.55.3.385
   Web of Science ®Google Scholar
• Worthington, D. L., & Schlottmann, R. S. (1986). The predictive validity of subtle and obvious empirically derived psychological test items under faking conditions. Journal of Personality Assessment, 50(2), 171–181. doi:10.1207/s15327752jpa5002_2
   PubMed Web of Science ®Google Scholar
• Wright, T. A., & Goodstein, J. (2007). Character is not “dead” in management research: A review of individual character and organizational-level virtue. Journal of Management, 33(6), 928–958. doi:10.1177/0149206307307644
   Web of Science ®Google Scholar
• Wright, T. A., & Lauer, T. L. (2013). What is character and why it really does matter. Organizational Dynamics, 42(1), 25–34. doi:10.1016/j.orgdyn.2012.12.004
   Web of Science ®Google Scholar
• Zavala, A. (1965). Development of the forced-choice rating scale technique. Psychological Bulletin, 63(2), 117–124. doi:10.1037/h0021567
   PubMed Web of Science ®Google Scholar