Planning Missing Data Designs for Human Ratings in Creativity Research: A Practical Guide

References

• Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & F. Csáki (Eds.), 2nd international symposium on information theory (pp. 267–281). Budapest, Hungary: Akadémiai Kiadó.
   Google Scholar
• Amabile, T. M. (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997–1013. doi:10.1037/0022-3514.43.5.997
   Web of Science ®Google Scholar
• Barbot, B. (2020). Creativity and self‐esteem in adolescence: A study of their domain‐specific, multivariate relationships. The Journal of Creative Behavior, 54(2), 279–292. doi:10.1002/jocb.365
   Web of Science ®Google Scholar
• Beaty, R. E., & Johnson, D. R. (2021). Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior Research Methods, 53(2), 757–780. doi:10.3758/s13428-020-01453-w
   PubMed Web of Science ®Google Scholar
• Beaty, R. E., & Silvia, P. (2013). Metaphorically speaking: Cognitive abilities and the production of figurative language. Memory & Cognition, 41(2), 255–267. doi:10.3758/s13421-012-0258-5
   PubMed Web of Science ®Google Scholar
• Benedek, M., Mühlmann, C., Jauk, E., & Neubauer, A. C. (2013). Assessment of divergent thinking by means of the subjective top-scoring method: Effects of the number of top-ideas and time-on-task on reliability and validity. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 341–349. doi:10.1037/a0033644
   PubMed Web of Science ®Google Scholar
• Buczak, P., Huang, H., Forthmann, B., & Doebler, P. (2022). The machines take over: A comparison of various supervised learning approaches for automated scoring of divergent thinking tasks. The Journal of Creative Behavior, 57(1), 17–36. doi:10.1002/jocb.559
   Web of Science ®Google Scholar
• Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6). doi:10.18637/jss.v048.i06
   Web of Science ®Google Scholar
• Christensen, P. R., Guilford, J. P., & Wilson, R. C. (1957). Relations of creative responses to working time and instructions. Journal of Experimental Psychology, 53(2), 82–88. doi:10.1037/h0045461
   PubMedGoogle Scholar
• Christensen, A. P., Silvia, P. J., Nusbaum, E. C., & Beaty, R. E. (2018). Clever people: Intelligence and humor production ability. Psychology of Aesthetics, Creativity, and the Arts, 12(2), 136–143. doi:10.1037/aca0000109
   Web of Science ®Google Scholar
• Cicchetti, D. V. (2001). Methodological commentary the precision of reliability and validity estimates re-visited: Distinguishing between clinical and statistical significance of sample size requirements. Journal of Clinical and Experimental Neuropsychology, 23(5), 695–700. doi:10.1076/jcen.23.5.695.1249
   PubMed Web of Science ®Google Scholar
• Dumas, D., Organisciak, P., & Doherty, M. (2020). Measuring divergent thinking originality with human raters and text-mining models: A psychometric comparison of methods. Psychology of Aesthetics, Creativity, and the Arts, 15(4), 645–663. doi:10.1037/aca0000319
   Web of Science ®Google Scholar
• Ferrando, P. J., & Lorenzo-Seva, U. (2018). Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educational and Psychological Measurement, 78(5), 762–780. doi:10.1177/0013164417719308
   PubMed Web of Science ®Google Scholar
• Forthmann, B., Holling, H., Zandi, N., Gerwig, A., Çelik, P., Storme, M., & Lubart, T. (2017). Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129–139. doi:10.1016/j.tsc.2016.12.005
   Web of Science ®Google Scholar
• Fürst, G. (2020). Measuring creativity with planned missing data. The Journal of Creative Behavior, 54(1), 150–164. doi:10.1002/jocb.352
   Web of Science ®Google Scholar
• Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60(1), 549–576. doi:10.1146/annurev.psych.58.110405.085530
   PubMed Web of Science ®Google Scholar
• Hass, R. W., Rivera, M., & Silvia, P. J. (2018). On the dependability and feasibility of layperson ratings of divergent thinking. Frontiers in Psychology, 9, 9. doi:10.3389/fpsyg.2018.01343
   PubMed Web of Science ®Google Scholar
• Kaufman, J. C., Baer, J., Cropley, D. H., Reiter-Palmon, R., & Sinnett, S. (2013). Furious activity vs. Understanding: How much expertise is needed to evaluate creative work? Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332–340. doi:10.1037/a0034809
   Web of Science ®Google Scholar
• Kleinkorres, R., Forthmann, B., & Holling, H. (2021). An experimental approach to investigate the involvement of cognitive load in divergent thinking. Journal of Intelligence, 9(1), 3. doi:10.3390/jintelligence9010003
   PubMed Web of Science ®Google Scholar
• Kornilov, S. A., Kornilova, T. V., & Grigorenko, E. L. (2016). The cross-cultural invariance of creative cognition: A case study of creative writing in U.S. and Russian college students. New Directions for Child and Adolescent Development, 2016(151), 47–59. doi:10.1002/cad.20149
   PubMedGoogle Scholar
• Long, H. (2014). More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks. Thinking Skills and Creativity, 13, 183–194. doi:10.1016/j.tsc.2014.05.002
   Web of Science ®Google Scholar
• Long, H., & Pang, W. (2015). Rater effects in creativity assessment: A mixed methods investigation. Thinking Skills and Creativity, 15, 13–25. doi:10.1016/j.tsc.2014.10.004
   Web of Science ®Google Scholar
• Matlock, K. L., Turner, R. C., & Gitchel, W. D. (2018). A study of reverse-worded matched item Pairs using the generalized partial credit and nominal response models. Educational and Psychological Measurement, 78(1), 103–127. doi:10.1177/0013164416670211
   PubMed Web of Science ®Google Scholar
• Mouchiroud, C., & Lubart, T. (2001). Children’s original thinking: An empirical examination of alternative measures derived from divergent thinking tasks. The Journal of Genetic Psychology, 162(4), 382–401. doi:10.1080/00221320109597491
   PubMed Web of Science ®Google Scholar
• Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1), i–30. doi:10.1002/j.2333-8504.1992.tb01436.x
   Google Scholar
• Myszkowski, N. (2021). Development of the R library “jrt”: Automated item response theory procedures for judgment data and their application with the consensual assessment technique. Psychology of Aesthetics, Creativity, and the Arts, 15(3), 426–438. doi:10.1037/aca0000287
   Web of Science ®Google Scholar
• Myszkowski, N., & Storme, M. (2019). Judge response theory? A call to upgrade our psychometrical account of creativity judgments. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 167–175. doi:10.1037/aca0000225
   Web of Science ®Google Scholar
• Nusbaum, E. C., Silvia, P. J., & Beaty, R. E. (2017). Ha ha? Assessing individual differences in humor production ability. Psychology of Aesthetics, Creativity, and the Arts, 11(2), 231–241. doi:10.1037/aca0000086
   Web of Science ®Google Scholar
• Plucker, J. A., Beghetto, R. A., & Dow, G. T. (2004). Why isn’t creativity more important to educational psychologists? Potentials, pitfalls, and future directions in creativity research. Educational Psychologist, 39(2), 83–96. doi:10.1207/s15326985ep3902_1
   Web of Science ®Google Scholar
• Primi, R. (2014). Divergent productions of metaphors: Combining many-facet rasch measurement and cognitive psychology in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 8(4), 461–474. doi:10.1037/a0038055
   Web of Science ®Google Scholar
• Primi, R., Silvia, P. J., Jauk, E., & Benedek, M. (2019). Applying many-facet Rasch modeling in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 176–186. doi:10.1037/aca0000230
   Web of Science ®Google Scholar
• R Core Team. (2021). R: A language and environment for statistical computing (4.1.2). Vienna, Austria: R Foundation for Statistical Computing.
   Google Scholar
• Reinig, B. A., & Briggs, R. O. (2013). Putting quality first in ideation research. Group Decision and Negotiation, 22(5), 943–973. doi:10.1007/s10726-012-9338-y
   Web of Science ®Google Scholar
• Robitzsch, A., & Steinfeld, J. (2018). Item response models for human ratings: Overview, estimation methods, and implementation in R. Psychological Test and Assessment Modeling, 60(1), 101–138.
   Google Scholar
• Runco, M. A., Plucker, J. A., & Lim, W. (2001). Development and psychometric integrity of a measure of ideational behavior. Creativity Research Journal, 13(3–4), 393–400. doi:10.1207/S15326934CRJ1334_16
   Web of Science ®Google Scholar
• Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1), 1–97. doi:10.1007/BF03372160
   Google Scholar
• Shaw, A. (2021). It works … but can we make it easier? A comparison of three subjective scoring indexes in the assessment of divergent thinking. Thinking Skills and Creativity, 40, 100789. doi:10.1016/j.tsc.2021.100789
   Web of Science ®Google Scholar
• Signorell, A. (2021). DescTools: Tools for descriptive statistics (R package version 0.99.44).
   Google Scholar
• Silvia, P. J., Kwapil, T. R., Walsh, M. A., & Myin-Germeys, I. (2014). Planned missing-data designs in experience-sampling research: Monte Carlo simulations of efficient designs for assessing within-person constructs. Behavior Research Methods, 46(1), 41–54. doi:10.3758/s13428-013-0353-y
   PubMed Web of Science ®Google Scholar
• Silvia, P. J., Martin, C., & Nusbaum, E. C. (2009). A snapshot of creativity: Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4(2), 79–85. doi:10.1016/j.tsc.2009.06.005
   Web of Science ®Google Scholar
• Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T. … Richard, C. A. (2008). Assessing creativity with divergent thinking tasks: Exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts, 2(2), 68–85. doi:10.1037/1931-3896.2.2.68
   Google Scholar
• Stevenson, C. E., Smal, I., Baas, M., Dahrendorf, M., Grasman, R. & van der Maas, H. (2020). Automated AUT scoring using a big data variant of the consensual assessment technique.
   Google Scholar
• Storme, M., Myszkowski, N., Çelik, P., & Lubart, T. (2014). Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32, 19–25. doi:10.1016/j.lindif.2014.03.002
   Web of Science ®Google Scholar
• Taylor, C. L., & Barbot, B. (2021). Dual pathways in creative writing processes. Psychology of Aesthetics, Creativity, and the Arts. doi:10.1037/aca0000415
   PubMed Web of Science ®Google Scholar
• Wagenmakers, E.-J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic Bulletin & Review, 11(1), 192–196. doi:10.3758/BF03206482
   PubMed Web of Science ®Google Scholar
• Wilson, R. C., Guilford, J. P., & Christensen, P. R. (1953). The measurement of individual differences in originality. Psychological Bulletin, 50(5), 362–370. doi:10.1037/h0060857
   PubMed Web of Science ®Google Scholar