Using Generalizability Theory to Disattenuate Correlation Coefficients for Multiple Sources of Measurement Error

References

• Ark, T. K. (2015). Ordinal generalizability theory using an underlying latent variable framework. (Doctoral dissertation). University of British Columbia, Vancouver, Canada. doi:10.14288/1.0166304
   Google Scholar
• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. doi:10.18637/jss.v067.i01
   Web of Science ®Google Scholar
• Becker, G. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5, 370–379. doi:10.1037/1082-989X.5.3.370
   PubMed Web of Science ®Google Scholar
• Brennan, R. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24, 339–353. doi:10.1177/01466210022031796
   Web of Science ®Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Brennan, R. L. (2010). Generalizability theory and classical test theory. Applied Measurement in Education, 24, 1–21, doi:10.1080/08957347.2011.532417
   Web of Science ®Google Scholar
• Brennan, R. L., & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277–289. doi:10.1111/j.1745-3984.1977.tb00045.x
   Web of Science ®Google Scholar
• Brown, W. (1910). Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3, 296–322. doi:10.1111/j.2044-8295.1910.tb00207.x
   Google Scholar
• Burt, C. (1936). The analysis of examination marks. In P. Hartog & E. C. Rhode (Eds.), The marks of examiners (pp. 245–314). London: Macmillan.
   Google Scholar
• Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). The Minnesota Multiphasic Personality Inventory-2 (MMPI-2): Manual for administration and scoring. Minneapolis, MN: University of Minnesota Press.
   Google Scholar
• Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. New York, NY: Routledge.
   Google Scholar
• Cattell, R. B., Cattell, A. K., & Cattell, H. E. P. (1993). 16PF Fifth edition questionnaire. Champaign, IL: Institute for Personality and Ability Testing.
   Google Scholar
• Chmialewski, M., & Watson, D. (2009). What is being assessed and why it matters: The impact of transient error in trait research. Journal of Personality and Social Psychology, 97, 186–202. doi:10.1037/a0015618
   PubMed Web of Science ®Google Scholar
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt.
   Google Scholar
• Cronbach, L. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12, 1–16. doi:10.1007/BF02289289
   PubMed Web of Science ®Google Scholar
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334. doi:10.1007/BF02310555
   Web of Science ®Google Scholar
• Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
   Google Scholar
• Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373–399. doi:10.1177/0013164497057003001
   Web of Science ®Google Scholar
• Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163. doi:10.1111/j.2044-8317.1963.tb00206.x
   Web of Science ®Google Scholar
• Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391–418. Retrieved from doi:10.1177/0013164404266386
   Web of Science ®Google Scholar
• Davey, T., Ferrara, S., Holland, P. W., Shavelson, R., Webb, N. M., & Wise, L. L. (2015). Psychometric considerations for the next generation of performance assessment. Princeton, NJ: Educational Testing Service.
   Google Scholar
• De Vries, R. E., Zettler, I., & Hilbig, B. E. (2014). Rethinking trait conceptions of social desirability scales: Impression management as an expression of honesty-humility. Assessment, 21, 286–299. doi:10.1177/1073191113504619
   PubMed Web of Science ®Google Scholar
• Dunbar, S. B., Koretz, D. M., & Hoover, H. D. (1991). Quality control in the development and use of performance assessments. Applied Measurement in Education, 4, 289–303. doi:10.1207/s15324818ame0404_3
   Google Scholar
• Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399–412. doi:10.1111/bjop.12046
   PubMed Web of Science ®Google Scholar
• Ebel, R. L. (1951). Estimation of reliability of ratings. Psychometrika, 19, 407–424. doi:10.1007/BF02288803
   Google Scholar
• Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 105–146). New York, NY: Macmillan.
   Google Scholar
• Flanagan, J. C. (1937). A proposed procedure for increasing the efficiency of objective tests. Journal of Educational Psychology, 26, 17–21. doi:10.1037/h0057430
   Google Scholar
• Geiser, C., Keller, B. T., Lockhart, G., Eid, M., Cole, D. A., & Koch, T. (2015). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses. Behavior Research Methods, 47, 172–203. doi:10.3758/s13428-014-0457-z
   PubMed Web of Science ®Google Scholar
• Gleser, G. C., Cronbach, L. J., & Rajaratnam, N. (1965). Generalizability of scores influenced by multiple sources of variance. Psychometrika, 30, 395–418. doi:10.1007/BF02289531
   PubMed Web of Science ®Google Scholar
• Goldman, B. A., Mitchel, D. F., & Egelson, P. E. (1997). Directory of unpublished mental measures (Vol. 7). Washington, DC: American Psychological Association.
   Google Scholar
• Green, S. B., & Yang, Y. (2009). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74, 121–135. doi:10.1007/s11336-008-9098-4
   Web of Science ®Google Scholar
• Gonzalez, O., & MacKinnon, D. P. (2018). A bifactor approach to model multifaceted constructs in statistical analysis. Educational and Psychological Measurement, 78, 5–31. doi:10.1177/0013164416673689
   PubMed Web of Science ®Google Scholar
• Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65–110). Wesport, CT: American Council on Education.
   Google Scholar
• Hogan, T. P., Benjamin, A., & Brezinski, K. L. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523–531. doi:10.1177/00131640021970691
   Web of Science ®Google Scholar
• Hoyt, C. J. (1941). Test reliability estimated by analysis of variance. Psychometricka, 48, 153–160. doi:10.1007/BF02289270
   Google Scholar
• Hoyt, W. T., & Melby, J. N. (1999). Dependability of measurement in counseling psychology: An introduction to generalizability theory. The Counseling Psychologist, 27, 325–352. doi:10.1177/0011000099273003
   Web of Science ®Google Scholar
• Huang, C. (2013). Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance. European Journal of Psychology of Education, 28, 663–683. doi:10.1007/s10212-012-0134-5
   Web of Science ®Google Scholar
• Jackson, D. N. (1984). Personality Research Form manual (3rd ed.). Port Huron, MI: Research Psychologists Press.
   Google Scholar
• John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big Five Inventory—Versions 4a and 54. Berkeley: University of California, Berkeley, Institute of Personality and Social Research.
   Google Scholar
• Lane, S., Liu, M., Ankenmann, R. D., & Stone, C. A. (1996). Generalizability and validity of a mathematics performance assessment. Journal of Educational Measurement, 33, 71–92. doi:10.1111/j.1745-3984.1996.tb00480.x
   Web of Science ®Google Scholar
• Le, H., Schmidt, F. L., & Putka, D. J. (2009). The multifaceted nature of measurement artifacts and its implications for estimating construct level relationships. Organizational Research Methods, 12, 165–200. doi:10.1177/1094428107302900
   Web of Science ®Google Scholar
• Linn, R. L., & Burton, E. (1994). Performance-based assessment: Implications of task specificity. Educational Measurement: Issues and Practice, 13(1), 5–15. doi:10.1111/j.1745-3992.1994.tb00778.x
   Google Scholar
• Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
   Google Scholar
• Lindquist, E. F. (1953). Design and analysis of experiments in psychology and education. Boston, MA: Houghton Mifflin.
   Google Scholar
• Marcoulides, G. A. (1990). An alternative method for estimating variance components in generalizability theory. Psychological Reports, 66, 379–386. doi:10.2466/pr0.1990.66.2.379
   Web of Science ®Google Scholar
• Marsh, H. W. (1992). Self-Description Questionnaire (SDQ) III: A theoretical and empirical basis for the measurement of multiple dimensions of late adolescent self-concept. An interim test manual and research monograph. Macarthur, New South Wales: University of Western Sydney, Faculty of Education.
   Google Scholar
• McCrae, R. R., & Costa, P. T., Jr. (2010). NEO inventories professional manual. Odessa, FL: Psychological Assessment Resources.
   Google Scholar
• McCrae, R. R. (2015). A more nuanced view of reliability: Specificity in the trait hierarchy. Personality and Social Psychology Review, 19, 97–112. doi:10.1177/1088868314541857
   PubMed Web of Science ®Google Scholar
• McCrae, R. R., Kurtz, J. E., Yamagata, S., & Terracciano, A. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28–50. doi:10.1177/1088868310366253
   PubMed Web of Science ®Google Scholar
• McDonald, R. P. (1999). Test theory: A unified approach. Mahwah, NJ: Erlbaum.
   Google Scholar
• Morris, C. A., Vispoel, W. P., & Kilinc, M. (2017, April). A latent state-trait theory approach to deriving reliability coefficients for congeneric measures. Paper presented at the annual meeting of the National Council on Measurement in Education, San Antonio, TX.
   Google Scholar
• Mushquash, C., & O'Connor, B. P. (2006). SPSS and SAS programs for generalizability theory analyses. Behavior Research Methods, 38, 542–547. doi:10.3758/BF03192810
   PubMed Web of Science ®Google Scholar
• Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press. (Measures of social psychological attitudes series)
   Google Scholar
• Paulhus, D. L. (1999). Paulhus Deception Scales (PDS): The Balanced Inventory of Desirable Responding-7. User's manual. Toronto: Multi-Health Systems.
   Google Scholar
• Paulhus, D. L., & Trapnell, P. D. (2008). Self-presentation: An agency-communion framework. In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality psychology: Theory and research (3rd ed., pp. 492–517). New York, NY: Guilford Press.
   Google Scholar
• Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., & R Core Team. (2016). Nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1-128. Retrieved from http://CRAN.R-project.org/package=nlme.
   Google Scholar
• Rajaratnam, N., Cronbach, L. J., & Gleser, G. C. (1965). Generalizability of stratified-parallel tests. Psychometrika, 30, 39–56. doi:10.1007/BF02289746
   PubMed Web of Science ®Google Scholar
• Raju, N. S. (1977). A generalization of coefficient alpha. Psychometrika, 42, 549–565. doi:10.1007/BF02295978
   Web of Science ®Google Scholar
• Raykov, T., & Marcoulides, G. A. (2006). Estimation of generalizability coefficients via a structural equation modeling approach to scale reliability evaluation. International Journal of Testing, 6, 81–95. doi:10.1207/s15327574ijt0601_5
   Google Scholar
• Reeve, C. L., Heggestad, E. D., & George, E. (2005). Estimation of transient error in cognitive ability scales. International Journal of Selection and Assessment, 13, 316–320. doi:10.1111/j.1468-2389.2005.00328.x
   Web of Science ®Google Scholar
• Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667–696. doi:10.1080/00273171.2012.715555
   PubMed Web of Science ®Google Scholar
• Revelle, W. (2016). Psych: Procedures for personality and psychological research (1.6.4). [software package and manual]. Evanston, IL: Northwestern University. Retrieved from https://cran.r-project.org/web/packages/psych
   Google Scholar
• Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016a). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98, 223–237. doi:10.1080/00223891.2015.1089249
   PubMed Web of Science ®Google Scholar
• Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016b). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137–150. doi:10.1037/met0000045
   PubMed Web of Science ®Google Scholar
• Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1–36. doi:10.18637/jss.v048.i02
   Web of Science ®Google Scholar
• Rulon, P. J. (1939). A simplified procedure for determining the reliability of a test by split-halves. Harvard Educational Review, 9, 99–103.
   Web of Science ®Google Scholar
• Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66, 507–514. doi:10.1007/BF02296192
   Web of Science ®Google Scholar
• Schmidt, F. L., & Hunter, J. E. (1999). Theory testing and measurement error. Intelligence, 27, 183–198. doi:10.1016/S0160-2896(99)00024-0
   Web of Science ®Google Scholar
• Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical investigation of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8, 206–224. doi:10.1177/1073191117700267
   PubMed Web of Science ®Google Scholar
• Shavelson, R. J., Baxter, G. P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30, 215–232. doi:10.1111/j.1745-3984.1993.tb00424.x
   Web of Science ®Google Scholar
• Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA: Sage. (Measurement methods for the social sciences)
   Google Scholar
• Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107–120. doi:10.1007/s11336-008-9101-0
   PubMed Web of Science ®Google Scholar
• Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101. doi:10.1093/ije/dyq191
   Web of Science ®Google Scholar
• Spearmen, C. (1910). Correlation calculated from faulty data. British Journal of Psychology, 3, 271–295. doi:10.1111/j.2044-8295.1910.tb00206.x
   Google Scholar
• Steyer, R., Ferrig, D., & Schmitt, M. (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8, 79–98.
   Google Scholar
• Steyer, R., Mayer, A., Geiser, C., & Cole, D. A. (2015). A theory of states and traits-revised. Annual Review of Clinical Psychology, 11, 71–98. doi:10.1146/annurev-clinpsy-032813-153719
   PubMed Web of Science ®Google Scholar
• Stöber, J., Dette, D. E., & Musch, J. (2002). Comparing continuous and dichotomous scoring of the Balanced Inventory of Desirable Responding. Journal of Personality Assessment, 78, 370–389. doi:10.1207/S15327752JPA7802_10
   PubMed Web of Science ®Google Scholar
• Strong, E. K., Jr., Donnay, D. A. C., Morris, M. L., Schaubhut, N. A., & Thompson, R. C. (2004). Strong Interest Inventory®, Revised Edition. Mountain View, CA: Consulting Psychologists Press, Inc.
   Google Scholar
• Thorndike, R. L. (1951). Reliability. In E. F. Lindquist (Ed.), Educational measurement (pp. 560–620). Washington, DC: American Council on Education. Retrieved from https://archive.org/details/educationalmeasu00lind
   Google Scholar
• Vispoel, W. P., & Forte Fast, E. E. (2000). Response biases and their relation to sex differences in multiple domains of self-concept. Applied Measurement in Education, 13, 79–97. doi:10.1207/s15324818ame1301_4
   Web of Science ®Google Scholar
• Vispoel, W. P., Kilinc, M., Morris, C. A., & Zhang, M. (2018, August). Extending the bifactor model to account for multiple sources of measurement error. Poster to be presented at the 2018 meeting of the American Psychological Association.
   Google Scholar
• Vispoel, W. P., & Kim, H. Y. (2014). Psychometric properties for the Balanced Inventory of Desirable Responding: Dichotomous versus polytomous conventional and IRT scoring. Psychological Assessment, 26, 878–891. doi:10.1037/a0036430
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Kilinc, M. (2018a). Applications of generalizability theory and their relations to classical test theory and structural equation modeling. Psychological Methods, 23, 1–26. doi:10.1037/met0000107
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Kilinc, M. (2018b). Practical applications of generalizability theory for designing, evaluating, and improving psychological assessments. Journal of Personality Assessment, 100, 53–67. doi:10.1080/00223891.2017.1296455
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Kilinc, M. (2018c). Using G-theory to enhance evidence of reliability and validity for common uses of the Paulhus Deception Scales. Assessment, 25, 69–83. doi:10.1177/1073191116641182
   PubMed Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Kilinc, M. (in press). Using generalizability theory with continuous latent response variables. Psychological Methods.
   Web of Science ®Google Scholar
• Vispoel, W. P., Morris, C. A., & Sun, L. (in press). Computerized and traditional administration of questionnaires: Psychometric quality and completion time for measures of self-concept. Journal of Experimental Education.
   Web of Science ®Google Scholar
• Vispoel, W. P., & Tao, S. (2013). A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. Psychological Assessment, 25, 94–104. doi:10.1037/a0029061
   PubMed Web of Science ®Google Scholar
• Warne, R. T., Lazo, M., Ramos, T., & Ritter, N. (2012). Statistical methods used on gifted journals, 2006–2010. Gifted Child Quarterly, 56, 134–149. doi:10.1177/0016986212444122
   Web of Science ®Google Scholar
• Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006). Reliability coefficients and generalizability theory. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 81–124). Amsterdam: Elsevier. doi:10.1016/S0169-7161(06)26004-8
   Google Scholar
• Wiley, E. W., Webb, N. M., & Shavelson, R. J. (2013). The generalizability of test scores. In K. F. Geisinger et al. (Eds.), APA handbook of testing and assessment in psychology: Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 43–60). Washington, DC: American Psychological Association.
   Google Scholar
• Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbach's α, Revelle β, and McDonald's ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70, 123–133. doi:10.1007/s11336-003-0974-7
   Web of Science ®Google Scholar