What’s Creative About Sentences? A Computational Approach to Assessing Creativity in a Sentence Generation Task

References

• Acar, S., & Runco, M. A. (2014). Assessing associative distance among ideas elicited by tests of divergent thinking. Creativity Research Journal, 26(2), 229–238. doi:10.1080/10400419.2014.901095
   Web of Science ®Google Scholar
• Acar, S., & Runco, M. A. (2019). Divergent thinking: New methods, recent research, and extended theory. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 153–158. doi:10.1037/aca0000231
   Web of Science ®Google Scholar
• Amabile, T. M. (1982). Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 997–1013. doi:10.1037/0022-3514.43.5.997
   Web of Science ®Google Scholar
• Ashton, S., & Davies, R. S. (2015). Using scaffolded rubrics to improve peer assessment in a MOOC writing course. Distance Education, 36(3), 312–334. doi :10.1080/01587919.2015.1081733
   Web of Science ®Google Scholar
• Barbot, B., Reiter-Palmon, R., Barbot, B., & Reiter-Palmon, R. (2019). Creativity assessment: Pitfalls, solutions, and standards [Special Issue]. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 131–132. doi:10.1037/aca0000251
   Web of Science ®Google Scholar
• Beaty, R. E., & Johnson, D. R. (2021). Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior Research Methods, 53(2), 757–780. doi:10.3758/s13428-020-01453-w
   PubMed Web of Science ®Google Scholar
• Beaty, R. E., Johnson, D. R., Zeitlen, D. C., & Forthmann, B. (2022). Semantic distance and the alternate uses task: Recommendations for reliable automated assessment of originality. Creativity Research Journal, 1–16. doi:10.1080/10400419.2022.2025720
   Web of Science ®Google Scholar
• Beketayev, K., & Runco, M. A. (2016). Scoring divergent thinking tests by computer with a semantics-based algorithm. Europe’s Journal of Psychology, 12(2), 210–220. doi:10.5964/ejop.v12i2.1127
   PubMed Web of Science ®Google Scholar
• Benedek, M., Fink, A., & Neubauer, A. C. (2006). Enhancement of ideational fluency by means of computer-based training. Creativity Research Journal, 18(3), 317–328. doi:10.1207/s15326934crj1803_7
   Web of Science ®Google Scholar
• Benedek, M., Nordtvedt, N., Jauk, E., Koschmieder, C., Pretsch, J., Krammer, G., & Neubauer, A. C. (2016). Assessment of creativity evaluation skills: A psychometric investigation in prospective teachers. Thinking Skills and Creativity, 21, 75–84. doi:10.1016/j.tsc.2016.05.007
   Web of Science ®Google Scholar
• Ceh, S. M., Annerer‐Walcher, S., Körner, C., Rominger, C., Kober, S. E., Fink, A., & Benedek, M. (2020). Neurophysiological indicators of internal attention: An electroencephalography–eye‐tracking coregistration study. Brain and Behavior, 10(10). doi:10.1002/brb3.1790
   PubMed Web of Science ®Google Scholar
• Ceh, S. M., Annerer-Walcher, S., Koschutnig, K., Körner, C., Fink, A., & Benedek, M. (2021). Neurophysiological indicators of internal attention: An fMRI–eye-tracking coregistration study. Cortex, 143, 29–46. doi:10.1016/j.cortex.2021.07.005
   PubMed Web of Science ®Google Scholar
• Ceh, S. M., Edelmann, C., Hofer, G., & Benedek, M. (2022). Assessing raters: What factors predict discernment in novice creativity raters? The Journal of Creative Behavior, 56(1), 41–54. doi:10.1002/JOCB.515
   Web of Science ®Google Scholar
• Cilibrasi, R., & Vitanyi, P. M. B. (2007). The google similarity distance. ArXiv:Cs/0412098. http://arxiv.org/abs/cs/0412098
   Google Scholar
• Corazza, G. E., Agnoli, S., & Mastria, S. (2022). The dynamic creativity framework: Theoretical and empirical investigations. European Psychologist. doi:10.1027/1016-9040/a000473
   Web of Science ®Google Scholar
• Cseh, G. M., & Jeffries, K. K. (2019). A scattered CAT: A critical evaluation of the consensual assessment technique for creativity research. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 159–166. doi:10.1037/aca0000220
   Web of Science ®Google Scholar
• Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint, (Vol. 1, pp. 4171–4186). doi:10.48550/arxiv.1810.04805
   Google Scholar
• Diedrich, J., Benedek, M., Jauk, E., & Neubauer, A. C. (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35–40. doi:10.1037/a0038688
   Web of Science ®Google Scholar
• D’Souza, R. (2021). What characterises creativity in narrative writing, and how do we assess it? Research findings from a systematic literature search. Thinking Skills and Creativity, 42, 100949. doi:10.1016/j.tsc.2021.100949
   Web of Science ®Google Scholar
• Dumas, D., Organisciak, P., & Doherty, M. (2021). Measuring divergent thinking originality with human raters and text-mining models: A psychometric comparison of methods. Psychology of Aesthetics, Creativity, and the Arts, 15(4), 645–663. doi:10.1037/aca0000319
   Web of Science ®Google Scholar
• Dumas, D., & Runco, M. A. (2018). Objectively scoring divergent thinking tests for originality: A re-analysis and extension. Creativity Research Journal, 30(4), 466–468. doi:10.1080/10400419.2018.1544601
   Web of Science ®Google Scholar
• Estes, Z., & Ward, T. B. (2002). The emergence of novel attributes in concept modification. Creativity Research Journal, 14(2), 149–156. doi:10.1207/S15326934CRJ1402_2
   Web of Science ®Google Scholar
• Explosion AI GmbH. (2022). SpaCy industrial-strength natural language processing in python. https://spacy.io/
   Google Scholar
• Field, A. P., & Wilcox, R. R. (2017). Robust statistical methods: A primer for clinical psychology and experimental psychopathology researchers. Behaviour Research and Therapy, 98, 19–38. doi:10.1016/J.BRAT.2017.05.013
   PubMed Web of Science ®Google Scholar
• Form, S. (2019). Reaching Wuthering heights with brave new words: The influence of originality of words on the success of outstanding best‐sellers. The Journal of Creative Behavior, 53(4), 508–518. doi:10.1002/jocb.230
   Web of Science ®Google Scholar
• Forthmann, B., Holling, H., Zandi, N., Gerwig, A., Çelik, P., Storme, M., & Lubart, T. (2017). Missing creativity: The effect of cognitive workload on rater (dis-)agreement in subjective divergent-thinking scores. Thinking Skills and Creativity, 23, 129–139. doi:10.1016/j.tsc.2016.12.005
   Web of Science ®Google Scholar
• Forthmann, B., Oyebade, O., Ojo, A., Günther, F., & Holling, H. (2019). Application of latent semantic analysis to divergent thinking is biased by elaboration. The Journal of Creative Behavior, 53(4), 559–575. doi:10.1002/jocb.240
   Web of Science ®Google Scholar
• Forthmann, B., Paek, S. H., Dumas, D., Barbot, B., & Holling, H. (2020). Scrutinizing the basis of originality in divergent thinking tests: On the measurement precision of response propensity estimates. British Journal of Educational Psychology, 90(3), e12325. doi:10.1111/bjep.12325
   Web of Science ®Google Scholar
• Fürst, G., Ghisletta, P., & Lubart, T. (2017). An experimental study of the creative process in writing. Psychology of Aesthetics, Creativity, and the Arts, 11(2), 202–215. doi:10.1037/ACA0000106
   Web of Science ®Google Scholar
• Glăveanu, V. (2015). Creativity as a sociocultural act. Journal of Creative Behavior, 49(3), 165–180. doi:10.1002/jocb.94
   Web of Science ®Google Scholar
• Guilford, J. P. (1967). The nature of human intelligence. McGraw-Hill.
   Google Scholar
• Hass, R. W. (2017). Tracking the dynamics of divergent thinking via semantic distance: Analytic methods and theoretical implications. Memory & Cognition, 45(2), 233–244. doi:10.3758/s13421-016-0659-y
   PubMed Web of Science ®Google Scholar
• Heinen, D. J. P., & Johnson, D. R. (2018). Semantic distance: An automated measure of creativity that is novel and appropriate. Psychology of Aesthetics, Creativity, and the Arts, 12(2), 144–156. doi:10.1037/aca0000125
   Web of Science ®Google Scholar
• Johnson, D. R., Kaufman, J. C., Baker, B. S., Barbot, B., Green, A. E., van Hell, J., … Beaty, R. E. (2022). Extracting creativity from narratives using distributional semantic modeling. Preprint. doi:10.31234/osf.io/fmwgy
   Google Scholar
• Kenett, Y. N. (2019). What can quantitative measures of semantic distance tell us about creativity? Current Opinion in Behavioral Sciences, 27, 11–16. doi:10.1016/j.cobeha.2018.08.010
   Web of Science ®Google Scholar
• Kettler, T., & Bower, J. (2017). Measuring creative capacity in gifted students: Comparing teacher ratings and student products. Gifted Child Quarterly, 61(4), 290–299. doi:10.1177/0016986217722617
   Web of Science ®Google Scholar
• Khalil, R., & Moustafa, A. A. (2022). A neurocomputational model of creative processes. Neuroscience and Biobehavioral Reviews, 137, 104656. doi:10.1016/j.neubiorev.2022.104656
   PubMed Web of Science ®Google Scholar
• Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2–3), 259–284. doi:10.1080/01638539809545028
   Web of Science ®Google Scholar
• Levenshtein, V. I. (1966). binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10, 707.
   Google Scholar
• MacLeod, C. M. (1991). Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin, 109(2), 163–203. doi:10.1037/0033-2909.109.2.163
   PubMed Web of Science ®Google Scholar
• Mastria, S., Agnoli, S., & Corazza, G. E. (2019). How does emotion influence the creativity evaluation of exogenous alternative ideas? PLoS ONE, 14(7), 1–16. doi:10.1371/journal.pone.0219298
   Web of Science ®Google Scholar
• Mednick, S. (1962). The associative basis of the creative process. Psychological Review, 69(3), 220–232. doi:10.1037/h0048850
   PubMed Web of Science ®Google Scholar
• Postel, H. J. (1969). Die Kölner Phonetik. Ein Verfahren zur Identifizierung von Personennamen auf der Grundlage der Gestaltanalyse. IBM-Nachrichten, 19, 925–931.
   Google Scholar
• Prabhakaran, R., Green, A. E., & Gray, J. R. (2014). Thin slices of creativity: Using single-word utterances to assess creative cognition. Behavior Research Methods, 46(3), 641–659. doi:10.3758/s13428-013-0401-7
   PubMed Web of Science ®Google Scholar
• Primi, R., Silvia, P. J., Jauk, E., & Benedek, M. (2019). Applying many-facet rasch modeling in the assessment of creativity. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 176–186. doi:10.1037/aca0000230
   Web of Science ®Google Scholar
• Ratcliff, J. W., & Metzener, D. E. (1988). Pattern matching: The gestalt approach. Dr Dobbs Journal, 13(7), 46.
   Google Scholar
• Reiter-Palmon, R., Forthmann, B., & Barbot, B. (2019). Scoring divergent thinking tests: A review and systematic framework. Psychology of Aesthetics, Creativity, and the Arts, 13(2), 144–152. doi:10.1037/aca0000227
   Web of Science ®Google Scholar
• Rothenberg, A. (1973). Word association and creativity. Psychological Reports, 33(1), 3–12. doi:10.2466/pr0.1973.33.1.3
   PubMed Web of Science ®Google Scholar
• Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92–96. doi:10.1080/10400419.2012.650092
   Web of Science ®Google Scholar
• Runco, M. A., Turkman, B., Acar, S., & Nural, M. (2017). Idea density and the creativity of written works. Journal of Genius and Eminence, 2(1), 26–31. doi:10.18536/jge.2017.04.02.01.03
   Google Scholar
• Schiller, A., Teufel, S., Thielen, C., & Stöckert, C. (1999). Guidelines für das Tagging deutscher Textcorpora mit STTS (Kleines und Großes Tagset). https://www.ims.uni-stuttgart.de/documents/ressourcen/lexika/tagsets/stts-1999.pdf
   Google Scholar
• Shapiro, R. J. (1970). The criterion problem. In P. E. Vernon (Ed.), Creativity (pp. 257–269). Penguin.
   Google Scholar
• Sharples, M. (1996). An account of writing as creative design. In C. Michael Levy and Sarah Ransdell (eds.), The Science of Writing. Theories, Methods, Individual Differences and Applications, 127–148. Mahwah: Erlbaum.
   Google Scholar
• Silvia, P. J. (2008). Discernment and creativity: How well can people identify their most creative ideas? Psychology of Aesthetics, Creativity, and the Arts, 2(3), 139–146. doi:10.1037/1931-3896.2.3.139
   Google Scholar
• Toubia, O., Berger, J., & Eliashberg, J. (2022). How quantifying the shape of stories predicts their success. Proceedings of the National Academy of Sciences, 118(26), e2011695118. doi:10.1073/pnas.2011695118
   Web of Science ®Google Scholar
• Turkman, B., & Runco, M. A. (2019). Discovering the creativity of written works: The keywords study. Gifted and Talented International, 34(1–2), 19–29. doi:10.1080/15332276.2019.1690955
   Google Scholar
• Wartena, C. (2019). A probabilistic morphology model for German lemmatization. Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019), Erlangen, Germany, (pp. 40–49). doi:10.25968/opus-1527
   Google Scholar
• Zedelius, C. M., Mills, C., & Schooler, J. W. (2019). Beyond subjective judgments: Predicting evaluations of creative writing from computational linguistic features. Behavior Research Methods, 51(2), 879–894. doi:10.3758/s13428-018-1137-1
   PubMed Web of Science ®Google Scholar