Practical Applications of Generalizability Theory for Designing, Evaluating, and Improving Psychological Assessments

References

• Ark, T. K. (2015). Ordinal generalizability theory using an underlying latent variable framework (Doctoral dissertation). Retrieved from https://open.library.ubc.ca/cIRcle/collections/ubctheses/24/items/1.0166304
   Google Scholar
• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. doi:10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Becker, G. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370–379. doi:10.1037/1082-989X.5.3.370
   PubMed Web of Science ®Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277–289. doi:10.1111/j.1745-3984.1977.tb00045.x
   Web of Science ®Google Scholar
• Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322. doi:10.1111/j.2044-8295.1910.tb00207.x
   Google Scholar
• Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. New York, NY: Routledge.
   Google Scholar
• Chmielewski, M., & Watson, D. (2009). What is being assessed and why it matters: The impact of transient error in trait research [Personality processes and individual differences]. Journal of Personality and Social Psychology, 97, 186–202. doi:10.1037/a0015618
   PubMed Web of Science ®Google Scholar
• Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
   Google Scholar
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart and Winston.
   Google Scholar
• Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163. doi:10.1111/j.2044-8317.1963.tb00206.x
   Web of Science ®Google Scholar
• de Vries, R. E., Zettler, I., & Hilbig, B. E. (2014). Rethinking trait conceptions of social desirability scales: Impression management as an expression of honesty-humility. Assessment, 21, 286–299. doi:10.1177/1073191113504619
   PubMed Web of Science ®Google Scholar
• Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399–412. doi:10.1111/bjop.12046
   PubMed Web of Science ®Google Scholar
• Geiser, C., Keller, B. T., Lockhart, G., Eid, M., Cole, D. A., & Koch, T. (2015). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses. Behavior Research Methods, 47, 172–203. doi:10.3758/s13428-014-0457-z
   PubMed Web of Science ®Google Scholar
• Gleser, G. C., Cronbach¸ L. J., & Rajaratnam, N. (1965). Generalizability of scores influenced by multiple sources of variance. Psychometrika, 30, 395–418. doi:10.1007/BF02289531
   PubMed Web of Science ®Google Scholar
• Graham, J. M. (2006). Congeneric and (essentially) tau-equivalent estimates of score reliability. Educational and Psychological Measurement, 66, 930–944. doi:10.1177/0013164406288165
   Web of Science ®Google Scholar
• Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121–135. doi:10.1007/s11336-008-9098-4
   Web of Science ®Google Scholar
• Green, S. B., Yang, Y., Alt, M., Brinkley, S., Gray, S., Hogan, T., & Cowan, N. (2016). Use of internal consistency coefficients for estimating reliability of experimental tasks. Psychonomic Bulletin and Review, 23, 750–763. doi:10.3758/s13423-015-0968-3
   PubMed Web of Science ®Google Scholar
• Hall, R. J., Snell, A. F., & Foust, M. S. (1999). Item parceling strategies in SEM: Investigating the subtle effects of unmodeled secondary constructs. Organizational Research Methods, 2, 233–256.
   Web of Science ®Google Scholar
• Huang, C. (2013). Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance. European Journal of Psychology of Education, 28, 663–683. doi:10.1007/s10212-012-0134-5
   Web of Science ®Google Scholar
• Le, H., Schmidt, F. L., & Putka, D. J. (2009). The multifaceted nature of measurement artifacts and its implications for estimating construct-level relationships. Organizational Research Methods, 12, 165–200. doi:10.1177/1094428107302900
   Web of Science ®Google Scholar
• Little, T. D., Cunningham, W. A., Shahar, G., & Widaman, K. F. (2002). To parcel or not to parcel: Exploring the question, weighing the merits. Structural Equation Modeling, 9, 151–173. doi:10.1207/S15328007SEM0902_1
   Web of Science ®Google Scholar
• Little, T. D., Rhemtulla, M., Gibson, K., & Schoemann, A. M. (2013). Why the items versus parcels controversy needn't be one. Psychological Methods, 18, 285–300. doi:10.1037/a0033266
   PubMed Web of Science ®Google Scholar
• Marcoulides, G. A. (1990). An alternative method for estimating variance components in generalizability theory. Psychological Reports, 66, 379–386. doi:10.2466/pr0.1990.66.2.379
   Web of Science ®Google Scholar
• Marcoulides, G. A. (2000, March). Generalizability theory: Advancements and implementations. Invited colloquium presented at the 22nd Language Testing Research Colloquium, Vancouver, BC, Canada.
   Google Scholar
• Marsh, H. W. (1992). Self-Description Questionnaire (SDQ) III: A theoretical and empirical basis for the measurement of multiple dimensions of late adolescent self-concept. An interim test manual and research monograph. Macarthur, New South Wales, Australia: University of Western Sydney, Faculty of Education.
   Google Scholar
• Marsh, H. W., Lüdtke, O., Nagengast, B., Morin, A. J., & Von Davier, M. (2013). Why item parcels are (almost) never appropriate: Two wrongs do not make a right—Camouflaging misspecification with item parcels in CFA models. Psychological Methods, 18, 257–284. doi:10.1037/a0032773
   PubMed Web of Science ®Google Scholar
• McCrae, R. R. (2015). A more nuanced view of reliability: Specificity in the trait hierarchy. Personality and Social Psychology Review, 19, 97–112. doi:10.1177/1088868314541857
   PubMed Web of Science ®Google Scholar
• McCrae, R. R., & Costa, P. T., Jr. (2010). NEO Inventories professional manual. Odessa, FL: Psychological Assessment Resources.
   Google Scholar
• McCrae, R. R., Kurtz, J. E., Yamagata, S., & Terracciano, A. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28–50. doi:10.1177/1088868310366253
   PubMed Web of Science ®Google Scholar
• McDonald, R. P. (1999). Test theory: A unified approach. Mahwah, NJ: Erlbaum.
   Google Scholar
• Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press.
   Google Scholar
• Paulhus, D. L., & Trapnell, P. D. (2008). Self-presentation: An agency–communion framework. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality psychology: Theory and research (3rd ed., pp. 492–517). New York, NY: Guilford Press.
   Google Scholar
• Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., & the R Core Team. (2016). nlme: Linear and nonlinear mixed effects models (3.1–128) [Software package and manual]. Retrieved from https://cran.r-project.org/web/package=nlme
   Google Scholar
• Rajaratnam, N., Cronbach, L. J., & Gleser, G. C. (1965). Generalizability of stratified-parallel tests. Psychometrika, 30, 39–56. doi:10.1007/BF02289746
   PubMed Web of Science ®Google Scholar
• Raju, N. S. (1977). A generalization of coefficient alpha. Psychometrika, 42, 549–565. doi:10.1007/BF02295978
   Web of Science ®Google Scholar
• Reeve, C. L., Heggestad, E. D., & George, E. (2005). Estimation of transient error in cognitive ability scales. International Journal of Selection and Assessment, 13, 316–320. doi:10.1111/j.1468-2389.2005.00328.x
   Web of Science ®Google Scholar
• Revelle, W. (2016). psych: Procedures for personality and psychological research (1.6.4) [Software package and manual]. Evanston, IL: Northwestern University. Retrieved from https://cran.r-project.org/web/package=psych
   Google Scholar
• Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. doi:10.18637/jss.v048.i02
   Web of Science ®Google Scholar
• Rulon, P. J. (1939). A simplified procedure for determining the reliability of a test by split-halves. Harvard Educational Review, 9, 99–103.
   Web of Science ®Google Scholar
• Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical investigation of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206–224. doi:10.1037/1082-989X.8.2.206
   PubMed Web of Science ®Google Scholar
• Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107–120. doi:10.1007/s11336-008-9101-0
   PubMed Web of Science ®Google Scholar
• Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101. doi:10.1093/ije/dyq191
   Web of Science ®Google Scholar
• Spearman, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295. doi:10.1111/j.2044-8295.1910.tb00206.x
   Google Scholar
• Steyer, R., Ferring, D., & Schmitt, M. J. (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8, 79–98.
   Google Scholar
• Steyer, R., Mayer, A., Geiser, C., & Cole, D. A. (2015). A theory of states and traits—revised. Annual Review of Clinical Psychology, 11, 71–98. doi:10.1146/annurev-clinpsy-032813-153719
   PubMed Web of Science ®Google Scholar
• Thorndike, R. L. (1951). Reliability. In E. F. Lindquist (Ed.), Educational measurement (pp. 560–620). Washington, DC: American Council on Education. Retrieved from https://archive.org/details/educationalmeasu00lind
   Google Scholar
• Vispoel, W. P., & Forte Fast, E. E. (2000). Response biases and their relation to sex differences in multiple domains of self-concept. Applied Measurement in Education, 13, 79–97. doi:10.1207/s15324818ame1301_4
   Web of Science ®Google Scholar
• Vispoel, W. P., & Kim, H. Y. (2014). Psychometric properties for the Balanced Inventory of Desirable Responding: Dichotomous versus polytomous conventional and IRT scoring. Psychological Assessment, 26, 878–891. doi:10.1037/a0036430
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Kilinc, M. (2016). Using G-theory to enhance evidence of reliability and validity for common uses of the Paulhus Deception Scales. Assessment. Advance online publication. doi:10.1177/1073191116641182
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Kilinc, M. (2017). Applications of generalizability theory and their relations to classical test theory and structural equation modeling. Psychological Methods. Advance online publication. doi:10.1037/met0000107
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., & Tao, S. (2013). A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. Psychological Assessment, 25, 94–104. doi:10.1037/a0029061
   PubMed Web of Science ®Google Scholar
• Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach's α, Revelle β, and McDonald's ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123–133. doi:10.1007/s11336-003-0974-7
   Web of Science ®Google Scholar