Test Length and Decision Quality in Personnel Selection: When Is Short Too Short?

REFERENCES

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . 1985 . Standards for educational and psychological testing , Washington , DC : American Educational Research Association .
   Google Scholar
• Arthur , W. and Day , D. V. 1994 . Development of a short-form for the Raven Advanced Progressive Matrices . Educational and Psychological Measurement , 54 : 394 – 403 . doi: 10.1177/ 0013164494054002013
   Web of Science ®Google Scholar
• Arvey , R. D. and Renz , G. L. 1992 . Fairness in the selection of employees . Journal of Business Ethics , 11 : 331 – 340 . doi: 10.1007/BF00870545
   Web of Science ®Google Scholar
• Baron , H. and Austin , J. 2000 . Measuring ability via the Internet: Opportunities and issues Paper presented at the Annual Conference of the Society for Industrial and Organizational Psychology, New Orleans, LA
   Google Scholar
• Bartram , D. and Hambleton , R. K. 2006 . Computer-based testing and the internet: Issues and advances , West Sussex , , UK : John Wiley & Sons .
   Google Scholar
• Berk , R. A. 1986 . A consumer's guide to setting performance standards on criterion-referenced tests . Review of Educational Research , 56 : 137 – 172 . doi: 10.3102/00346543056001137
   Web of Science ®Google Scholar
• Born , M. and Scholarios , D. 2005 . “ Decision making in selection ” . In The Blackwell handbook of selection , Edited by: Evers , A. , Anderson , N. and Smit-Voskuijl , O. O. 267 – 290 . Malden , MA : Blackwell Publishing .
   Google Scholar
• Borsboom , D. , Romeijn , J. and Wicherts , J. M. 2008 . Measurement invariance versus selection invariance: Is fair selection possible? . Psychological Methods , 13 : 75 – 98 . doi: 10.1037/1082-989X.13.2.75
   PubMed Web of Science ®Google Scholar
• Brown , W. 1910 . Some experimental results in the correlation of mental abilities . British Journal of Psychology , 3 : 296 – 322 . doi: 10.1111/j.2044-8295.1910.tb00207.x
   Google Scholar
• Burisch , M. 1997 . Test length and validity revisited . European Journal of Personality , 11 : 303 – 315 . doi: 10.1002/(SICI)1099-0984(199711)11:4<303::AID-PER292>3.0.CO;2-#
   Web of Science ®Google Scholar
• Caldwell , D. F. and O’Reilly , C. A. III . 1990 . Measuring person-job fit with a profile-comparison process . Journal of Applied Psychology , 75 : 648 – 657 . doi: 10.1037/0021-9010.75.6.648
   Web of Science ®Google Scholar
• Campbell , J. P. 1979 . “ Problems and new directions for industrial psychology ” . In Handbook of industrial and organizational psychology , Edited by: Dunnette , M. D. 185 – 222 . Chicago , IL : Rand McNally College Publishing Company .
   Google Scholar
• Candell , G. L. and Drasgow , F. 1988 . An iterative procedure for linking metrics and assessing item bias in item response theory . Journal of Psychological Measurement , 12 : 253 – 260 . doi: 10.1177/014662168801200304
   Web of Science ®Google Scholar
• Cascio , W. F. 1980 . Responding to the demand for accountability: A critical analysis of three utility models . Organizational Behavior and Human Performance , 25 : 32 – 45 . doi: 10.1016/0030-5073(80)90024-0
   Google Scholar
• Charter , R. A. 2003 . A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability . The Journal of General Psychology , 130 : 290 – 304 . doi: 10.1080/00221300309601160
   PubMed Web of Science ®Google Scholar
• Cohen , A. S. , Kim , S. H. and Baker , F. B. 1993 . Detection of differential item functioning in the Graded Response Model . Applied Psychological Measurement , 17 : 335 – 350 . doi: 10.1177/014662169301700402
   Web of Science ®Google Scholar
• Cook , M. 2009 . Personnel selection: Adding value through people , (5th ed.) , London , UK : Wiley-Blackwell .
   Google Scholar
• Costa , P. T. Jr. and McCrae , R. R. 1992 . The NEO PI/FFI manual supplement , Odessa , FL : Psychological Assessment Resources .
   Google Scholar
• Cronbach , L. J. and Gleser , G. C. 1953 . Assessing similarity profiles . Psychological Bulletin , 50 : 456 – 473 .
   PubMed Web of Science ®Google Scholar
• Crowder , B. and Michael , W. B. 1991 . The development and validation of a short form of a multidimensional self-concept measure for high technology employees . Educational and Psychological Measurement , 51 : 447 – 454 . doi: 10.1177/0013164491512019
   Web of Science ®Google Scholar
• Dalen , L. H. , Stanton , N. A. and Roberts , A. D. 2001 . Faking personality questionnaires in personnel selection . The Journal of Management Development , 20 : 729 – 741 . doi: 10.1108/02621710110401428
   Google Scholar
• Dilchert , S. 2007 . Peaks and valleys: Predicting interests in leadership and managerial positions from personality profiles . International Journal of Selection and Assessment , 15 : 317 – 334 . doi: 10.1111/j.1468-2389.2007.00391.x
   Web of Science ®Google Scholar
• Drasgow , F. , Nye , C. D. and Tay , L. 2010 . “ Indicators of quality assessment ” . In Handbook of workplace assessment: Evidence-based practices for selecting and developing organizational talent , Edited by: Scott , J. C. and Reynolds , D. H. 27 – 60 . San Francisco , CA : Jossey-Bass .
   Google Scholar
• Edwards , J. R. 1993 . Problems with the use of profile similarity indexes in the study of congruence in organizational research . Personnel Psychology , 46 : 641 – 665 . doi: 10.1111/j.1744-6570.1993.tb00889.x
   Web of Science ®Google Scholar
• Embretson , S. E. and Reise , S. P. 2000 . Item response theory for psychologists , Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Emons , W. H. M. , Sijtsma , K. and Meijer , R. R. 2007 . On the consistency of individual classification using short scales . Psychological Methods , 12 : 105 – 120 . doi: 10.1037/1082-989x.12.1.105
   PubMed Web of Science ®Google Scholar
• Ercikan , K. and Julian , M. 2002 . Classification accuracy of assigning student performance to proficiency levels: Guidelines for assessment design . Applied Measurement in Education , 15 : 269 – 294 . doi: 10.1207/S15324818AME1503_3
   Web of Science ®Google Scholar
• Forsythe , C. , Grose , E. and Ratner , J. 1998 . Human factors and Web development , Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Fraley , R. C. , Waller , N. G. and Brennan , K. A. 2000 . An item response theory analysis of self-report measures of adult attachment . Journal of Personality and Social Psychology , 78 : 350 – 365 . doi: 10.1037/0022-3514.78.2.350
   PubMed Web of Science ®Google Scholar
• Gardner , D. G. , Cummings , L. L. , Dunham , R. B. and Pierce , J. L. 1998 . Single-item versus multiple-item measurement scales: An empirical comparison . Educational and Psychological Measurement , 58 : 898 – 915 . doi: 10.1177/0013164498058006003
   Web of Science ®Google Scholar
• Gardner , D. G. , Dunham , R. B. , Cummings , L. L. and Pierce , J. L. 1989 . Focus of attention at work: Construct definition and empirical validation . Journal of Occupational Psycholog , 62 : 61 – 77 .
   Web of Science ®Google Scholar
• Glaros , A. G. and Kline , R. B. 1988 . Understanding the accuracy of tests with cutting scores: The sensitivity, specificity, and predictive value model . Journal of Clinical Psychology , 44 : 1013 – 1023 . doi: 10.1002/1097-4679(198811)44:6<1013::AID-JCLP2270440627>3.0.CO;2-Z
   PubMed Web of Science ®Google Scholar
• Gosling , S. D. , Rentfrow , P. J. and Swann , W. B. Jr. 2003 . A very brief measure of the Big-Five personality domains . Journal of Research in Personality , 37 : 504 – 528 . doi: 10.1016/S0092-6566(03)00046-1
   Web of Science ®Google Scholar
• Guion , R. M. 1976 . “ Recruiting, selection, and job placement ” . In Handbook of industrial and organizational psychology , Edited by: Dunnette , M. D. 777 – 828 . Chicago , IL : Rand McNally College Publishing .
   Google Scholar
• Hambleton , R. K. and Slater , S. C. 1997 . Reliability of credentialing examinations and the impact of scoring models and standard-setting policies . Applied Measurement in Education , 10 : 19 – 38 . doi: 10.1207/s15324818ame1001_2
   Web of Science ®Google Scholar
• Harwell , M. R. and Janosky , J. E. 1991 . An empirical study of the effects of small datasets and varying prior variances on item parameter estimation in BILOG . Applied Psychological Measurement , 15 : 279 – 291 . doi: 10.1177/014662169101500308
   Web of Science ®Google Scholar
• Judge , T. A. , Erez , A. , Bono , J. E. and Thoresen , C. J. 2003 . The core self-evaluations scale: Development of a measure . Personnel Psychology , 56 : 303 – 331 . doi: 10.1111/j.1744-6570.2003.tb00152.x
   Web of Science ®Google Scholar
• Kline , P. 2000 . The handbook of psychological testing , (2nd ed.) , New York , NY : Routledge .
   Google Scholar
• Konradt , U. , Hertel , G. and Joder , K. 2003 . Web-based assessment of call center agents: Development and validation of a computerized instrument . International Journal of Selection and Assessment , 11 : 184 – 193 . doi: 10.1111/1468–2389.00242
   Web of Science ®Google Scholar
• Levy , P. 1968 . Short-form tests: A methodological review . Psychological Bulletin , 69 : 410 – 416 . doi: 10.1037/h0025736
   PubMed Web of Science ®Google Scholar
• Lord , F. M. and Novick , M. R. 1968 . Statistical theories of mental test scores , Reading , MA : Addison-Wesley .
   Google Scholar
• Lord , F. M. and Wingersky , M. S. 1984 . Comparison of IRT true-score and equipercentile observed-score “equatings . Applied Psychological Measurement , 8 : 453 – 461 . doi: 10.1177/ 014662168400800409
   Web of Science ®Google Scholar
• Martin , D. E. , Moore , C. F. , Sloan , L. and Legree , P. J. 2007 . Personnel selection: An application of the unobtrusive knowledge test . Journal of Business and Behavioral Sciences , 16 : 4 – 16 . Retrieved from http://ssrn.com/abstract=1739396
   Google Scholar
• Meehl , P. E. and Rosen , A. 1955 . Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores . Psychological Bulletin , 52 : 194 – 216 . doi: 10.1037/h0048070
   PubMed Web of Science ®Google Scholar
• Mosel , J. N. 1953 . Single-item tests for personnel screening . Educational and Psychological Measurement , 13 : 179 – 192 . doi: 10.1177/001316445301300204
   Web of Science ®Google Scholar
• Murphy , K. R. 1994 . Potential effects of banding as a function of test reliability . Personnel Psychology , 47 : 477 – 495 . doi: 10.1111/j.1744-6570.1994.tb01733.x
   Web of Science ®Google Scholar
• Netemeyer , R. G. , Pulling , C. and Bearden , W. O. 2003 . “ Observations on some key psychometric properties of paper-and-pencil measures ” . In Advances in business marketing and purchasing: Essays by distinguished scholars , Edited by: Woodside , A. G. and Moore , E. M. 115 – 138 . New York , NY : Elsevier Science .
   Google Scholar
• Reise , S. P. and Waller , N. G. 2009 . Item response theory and clinical measurement . Annual Review of Clinical Psychology , 5 : 27 – 48 . doi: 10.1146/annurev.clinpsy.032408.153553
   PubMed Web of Science ®Google Scholar
• Samejima , F. 1969 . Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17) , Richmond , VA : Psychometric Society . Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf
   Google Scholar
• Schmidt , F. L. and Hunter , J. E. 1998 . The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings . Psychological Bulletin , 124 : 262 – 274 . doi: 10.1037/0033-2909.124.2.262
   Web of Science ®Google Scholar
• Schmidt , F. L. , Hunter , J. E. , McKenzie , R. C. and Muldrow , T. W. 1979 . Impact of valid selection procedures on work-force productivity . Journal of Applied Psychology , 64 : 609 – 626 . doi: 10.1037/0021-9010.64.6.609
   Web of Science ®Google Scholar
• Segall , D. O. 2001 . General ability measurement: An application of multidimensional item response theory . Psychometrika , 66 : 79 – 97 . doi: 10.1007/BF02295734
   Web of Science ®Google Scholar
• Smith , G. T. , McCarthy , D. M. and Anderson , K. 2000 . On the sins of short-form development . Psychological Assessment , 12 : 102 – 111 . doi: 10.1037/1040-3590.12.1.102
   PubMed Web of Science ®Google Scholar
• Smith , M. and Smith , P. C. 2005 . Testing people at work: Competencies in psychometric testing , Malden , MA : BPS Blackwell .
   Google Scholar
• Smits , N. and Vorst , H. C. M. 2007 . Reducing the length of questionnaires through structurally incomplete designs: An illustration . Learning and Individual Differences , 17 : 25 – 34 . doi: 10.1016/j.lindif.2006.12.005
   Web of Science ®Google Scholar
• Spearman , C. 1910 . Correlation calculated from faulty data . British Journal of Psychology , 3 : 271 – 295 . doi: 10.1111/j.2044-8295.1910.tb00206.x
   Google Scholar
• Stanton , J. M. , Sinar , E. F. , Balzer , W. K. and Smith , P. C. 2002 . Issues and strategies for reducing the length of self-report scales . Personnel Psychology , 55 : 167 – 194 . doi: 10.1111/j.1744-6570.2002.tb00108.x
   Web of Science ®Google Scholar
• Taylor , H. C. and Russell , J. T. 1939 . The relationship of validity coefficients to the practical effectiveness of tests in selection: Discussion and tables . Journal of Applied Psychology , 23 : 565 – 578 . doi: 10.1037/h0057079
   Web of Science ®Google Scholar
• Thissen , D. , Pommerich , M. , Billeaud , K. and Williams , V. S. L. 1995 . Item response theory for scores including polytomous items with ordered responses . Applied Psychological Measurement , 19 : 39 – 49 . doi: 10.1177/014662169501900105
   Web of Science ®Google Scholar
• Van Iddekinge , C. H. and Ployhart , R. E. 2008 . Developments in the criterion-related validation of selection procedures: A critical review and recommendations for practice . Personnel Psychology , 61 : 871 – 925 . doi: DOI 10.1111/j.1744-6570.2008.00133.x
   Web of Science ®Google Scholar
• Wanous , J. P. and Hudy , M. J. 2001 . Single-item reliability: A replication and extension . Organizational Research Methods , 4 : 361 – 375 . doi: 10.1177/109442810144003
   Web of Science ®Google Scholar