The Patient is Thriving! Current Issues, Recent Advances, and Future Directions in Creativity Assessment

ABSTRACT

In 1998, Plucker and Runco provided an overview of creativity assessment, noting current issues (fluency confounds, generality vs. specificity), recent advances (predictive validity, implicit theories), and promising future directions (moving beyond divergent thinking measures, reliance on batteries of assessments, translation into practice). In the ensuing quarter century, the field experienced large growth in the quantity, breadth, and depth of assessment work, suggesting another analysis is timely. The purpose of this paper is to review the 1998 analysis and identify current issues, advances, and future directions for creativity measurement. Recent advances include growth in assessment quantity and quality and use of semantic distance as a scoring technique. Current issues include mismatches between current conceptions of creativity and those on which many measures are based, the need for psychometric quality standards, and a paucity of predictive validity evidence. The paper concludes with analysis of likely future directions, including use of machine learning to administer and score assessments and refinement of our conceptual frameworks for creativity assessment. Although the 1998 paper was written within an academic climate of harsh criticism of creativity measurement, the current climate is more positive, with reason for optimism about the continued growth of this important aspect of the field.

Across a number of fields, identifying and measuring creativity continues to grow in prevalence and importance. Within the private sector, businesses are focusing considerable resources on talent development, often with an emphasis on innovative skills. K-12 educators and policymakers seek ways to assess creativity to provide an alternative set of student outcomes that supplement traditional high-stakes testing. Many colleges and universities around the world have refocused their programs to help students develop innovative and entrepreneurial skills. And, of course, researchers across a number of fields rely on measures of creativity in their research and evaluation work.

Shortly before the turn of the previous century, many scholars noted that creativity assessment was under siege, with strident criticisms from both within (Baer, 1994; Cropley, 2000; Hunsaker & Callahan, 1995; Plucker & Renzulli, 1999) and outside the field (Eysenck, 1994; Houtz & Krug, 1995). Criticisms included a perceived lack of predictive validity evidence, a lack of content-specific measures at a time where context was being emphasized in education theory and research, overreliance on classical test theory and older measurement models, and an overreliance on divergent thinking tasks for assessment of creativity and creative potential. Supporters of assessment research granted the critics some of these points but noted that the critiques went too far or were simply incorrect.

Within this context, Plucker and Runco (1998) undertook a comprehensive review of creativity measurement, including both controversies and possible future directions. The paper was titled, “The death of creativity measurement has been greatly exaggerated,” because most public commentary on assessment of creativity had become harshly negative. Yet the paper’s overall conclusion was that important, recent advances had occurred in several aspects of creativity assessment, making them optimistic about the future of this aspect of the field.

A quarter century later, this optimism appears to have been warranted. There has been exponential growth in research and development on measurement related to creativity, entrepreneurship, and innovation, with several important lines of research on improvement of existing measures and creation of new assessment strategies. For this reason, revisiting the previous analysis is timely, as it provides the field with an opportunity to take stock of its recent progress, identify current and likely future trends, and note areas in need of additional effort.¹

From 1998 to now

Plucker and Runco (1998) offered three sets of observations: current issues being debated in the field, recent advances in creativity assessment, and future directions that could prove helpful to creativity measurement and the field in general. This section notes the extent to which subsequent research addressed the issues, continued or moved away from the trends, and matched the predictions.

Have we made progress on the field’s big assessment issues?

Two issues were noted as current issues in 1998, debates over domain generality and specificity and fluency as a contaminating factor in scoring of divergent thinking tasks. Regarding specificity-generality, many fields were incorporating sociocultural perspectives, with their emphasis on context and person-environment interaction (e.g., Bredo, 1994; O’Loughlin, 1992), at the turn of the century, and creativity scholarship was not an exception. Reacting to a strong domain-general view for much of the field’s history, Baer (1998) and others noted the need for strong domain specific theories and models with accompanying assessments.

The field’s emphasis on domain specificity became quite strong in ensuing years, but that pendulum has swung back toward the middle (again, as in many other fields). Although the evidence of content generality has been the subject of several recent studies (Qian & Plucker, 2018; Qian, Plucker, & Yang, 2019; van Broekhoven, Cropley, & Seegers, 2020), most models embrace both domain specific and domain general aspects of creativity. For example, the amusement park model of creativity proposes that creativity becomes more domain-specific as one moves from initial stages of the creative process to application in domains and micro-domains (Baer & Kaufman, 2005), and Plucker and Beghetto (2004) offered a conceptualization that emphasized the importance between a balance of domain general and domain specific attitudes, skills, and knowledge, with extremes in either direction likely hindering creativity (i.e., too much specificity leading to functional fixedness, and too much generality leading to superficiality).

Although this debate does not yet appear to be settled (and may never be!), scholars continue to put serious thought into the role of context and creativity. Regarding implications for assessment, Plucker, Meyer, and Makel (in press) suggest a developmental perspective: Assessing under an assumption of domain and task generality is reasonable for younger students, but domain specificity is appropriate for older students and adults, given the importance of the application of creativity to specific tasks and using specific content as one ages and moves into the workforce.

The idea of fluency of a contaminating factor in the scoring of performance on divergent thinking tasks goes back to at least Clark and Mirels (1970). They noted the high intercorrelations among creativity and intelligence measures, hypothesizing that fluency was the cause of these correlations. Intercorrelations dropped in magnitude considerably when fluency was controlled. Subsequently, several other researchers also noted the potentially confounding effects of fluency when scoring assessments, with several offering strategies for controlling for fluency effects (e.g., Feldhusen & Goh, 1995; Hocevar, 1979; Runco & Albert, 1985). Plucker and Runco (1998) recommended that the various strategies be examined in light of their impact on reliability and validity evidence, not dissimilar from the recent call by Barbot, Hass, and Reiter-Palmon (2019) for homogenization of measurement standards.

Although researchers attempted to address the role of fluency over the past couple decades (e.g., Plucker, Qian, & Schmalensee, 2014), the field moved far beyond these recommendations and undertook a wide-ranging analysis of new or improved scoring methods for DT task performance. Much of this work built on earlier efforts by Runco and colleagues (e.g., Runco & Mraz, 1992; Runco, Okuda, & Thurston, 1987), which was built upon by Silvia and colleagues in a number of important studies in the early 2000s (Silvia, 2011; Silvia, Martin, & Nusbaum, 2009; Silvia et al., 2008). Since then, the amount of DT scoring work has grown exponentially. Acar and Runco (2019) provide a comprehensive overview of the large number of studies on this topic.

Collectively, the sharp growth in DT scoring research is a very positive development for the field relative to a focus on fluency effects, given that (a) DT scoring in general needed fresh perspectives and (b) evidence suggests that fluency may have a genetic basis (i.e., it may be important in its own right and not “controlled for”; see Runco et al., 2011). Notably, Reiter-Palmon, Forthmann, and Barbot (2019) have proposed a framework for classifying the various aspects of scoring that need to be considered and reported in studies using DT-based assessments, including objectivity vs. subjectivity, individual item scoring vs. item pool scoring, and scoring dimension (fluency, originality, etc.).

Did recent advances continue to advance?

Two recent advances were studies of the predictive validity of assessments and increased sophistication in the study of implicit theories. Plucker and Runco noted that predictive validity of DT measures was generally assessed to be unimpressive (Baer, 1993; Wallach, 1976), but that the available evidence was more promising than many critics suggested, especially for domain-specific DT tasks (Hong, Milgram, & Gorsky, 1995; Okuda, Runco, & Berger, 1991; Sawyers & Canestaro, 1989). At the same time, they called for more such research, noting its importance to our understanding of creativity assessment and creative development across the lifespan.

Unfortunately, the ensuing quarter-century has seen few such studies. Torrance’s longitudinal study, which first collected data in 1958, has been reanalyzed and extended several times, with similarly positive conclusions about the predictive validity of the DT measures. For example, Plucker’s (1999) reanalysis of the Torrance data suggests childhood creativity predicted adult creative accomplishment, and that it did so at a higher level than intelligence test scores. Runco, Millar, Acar, and Cramond (2010) subsequently collected new data from the subjects in that study and found a positive if more complex pattern of longitudinal relationships. A meta-analysis of DT scores from students in grades 1–12 provides evidence that performance generally increases over time, but this study speaks more to development than predictive validity (Said-Metwaly, Fernández-Castilla, Kyndt, Van den Noortgate, & Barbot, 2021).

However, the predictive validity of other forms of creativity assessment is very limited. Kaufman et al. (2008) noted that critics were holding DT assessments to a predictive validity standard that was not being applied equally to other popular assessments (e.g., the consensual assessment technique). This situation has improved little since 1998 and 2008.

Implicit theories were a popular area of scholarship, with considerable innovation into the late 1990s on assessment of those theories and personal beliefs. This trend continued into the early 2000s (Puccio & Chimento, 2001), but the momentum appeared to slow considerably before increasing again in the mid-2010s. The type of implicit theory study also appears to have changed over the past two decades. In the earlier body of work, the majority of studies focused on differences in implicit theories of creativity among cultures (Chan & Chan, 1999; Lim & Plucker, 2001; Niu & Sternberg, 2002; Paletz & Peng, 2008; Panda & Yadava, 2005). Recent work has focused on implicit theories held by specific groups, such as teachers, professors, and students (Cropley, Patston, Marrone, & Kaufman, 2019; Gralewski & Karwowski, 2018; Maksić & Spasenović, 2018; Pavlović & Maksić, 2019). However, there are plentiful exceptions to these trends (e.g., Loewenstein & Mueller, 2016; Patston, Cropley, Marrone, & Kaufman, 2018; Spiel & von Korff, 1998).

However, the increase in quantity and sophistication of implicit theory studies does not appear to be matched with parallel development in the sophistication of assessment of those theories. For example, Malmelin and Nivari-Lindström (2017) assessed implicit theories by having journalists respond to a series of statements about the nature of creativity, which they then analyzed qualitatively to arrive at an understanding of how Finnish journalists conceptualize creative work. The study is unique in its highly applied, contextualized focus on professional journalists and its attention to details such as the use of “creative work” rather than “creativity” (see Runco, 2007). But the actual assessment strategy is very straightforward and not distinct from strategies used to assess implicit theories in the mid-1980s. Similarly, but using a more traditional quantitative approach for data analysis, Cropley et al. (2019) asked teachers to respond to a series of statements about creativity (e.g., engaging students’ creativity helps their learning, it is possible to measure student creativity, creative students are the ones who write, draw, or play music), with several items reverse-coded. They found support for five factors of teacher beliefs, namely that creativity is important for education, requires domain-specific knowledge, can be measured, is found in all subjects, and is not limited to certain students.

The recent research of implicit theories is interesting and help to build our knowledge base of how various groups of people think about creativity. But it is somewhat surprising that developments from the 1980s and 1990s regarding the assessment of implicit theories have not been incorporated or improved upon in the contemporary studies. For example, researchers from that earlier period found evidence that responses to close- and open-ended questions about creativity beliefs tended to reflect social desirability, but that when asked to use their implicit theories (such as in the evaluation of profiles of potentially creative students or employees), they appear to use different, more pragmatic conceptions of creativity (Lim & Plucker, 2001; Sternberg, 1985). The majority of recent studies on this topic appear to use only participant self-responses. The assessment of implicit theories of creativity appears to have become much broader regarding types of participants and their roles; but it has not become deeper over the past quarter century and may, in fact, have become less sophisticated.

Were the future directions fruitful?

The 1998 analysis also recommended three future directions for creativity assessment: broadened application of psychometric methods, reliance on batteries of assessments, and translation into practice. Regarding psychometric methods, few published studies were conducted from the perspective of modern measurement theories and techniques, such as item response theory, and the majority of studies did not use more sophisticated statistical methods, such as structural equation modeling. The field has moved in this direction, with more sophisticated, contemporary measurement strategies (e.g., Becker, Cabeza, Shaw, & Davidson, 2022; Primi, Silvia, Jauk, & Benedek, 2019; Qian & Plucker, 2018; Van Hooijdonk, Mainhard, Kroesbergen, & Van Tartwijk, 2022; Wang & Long, 2022). This work includes assessment of group creativity (Han, Long, Ge, & Pang, 2022; Hester & Hester, 2012; Romero et al., 2019), which is also being pursued by the Schmidt Futures Foundation.

This sophistication is also evident in conceptual analyses (Cerrato, Siano, Marco, & Ricci, 2019; Kaufman, 2019), such as reviews of research on the consensual assessment technique (Cseh & Jefferies, 2019; Myszkowski & Storme, 2019). In particular, Myszkowski and Storme propose the use of item response theory models to improve the CAT’s psychometric framework. Continuing to apply advances in measurement theory and technique will similarly advance development of the field’s creativity measures.

The growth of systems and sociocultural approaches to creativity led Plucker and Runco to recommend that creativity researchers use multiple types of measures when studying creativity.² There is little evidence of reliance on batteries of assessments in creativity studies over the past 25 years. For example, in the most recent issues of three major journals for the scientific study of creativity, 11 empirical articles included the assessment of creativity. Only six relied on more than one type of measure, with only one using more than two distinct assessment types. The exception here proves the rule: Ceh, Edelmann, Hofer, and Benedek (2022), in an examination of factors that correlate with novice raters’ ability to discern more and less creative responses to DT tasks, used divergent thinking tasks, personality measures, an activity checklist, and evaluation of others’ creativity. The authors conclude that DT performance, past creative activities, and openness to experience (among other factors) predict discernment. By using a broad and rather comprehensive set of assessment types, the researchers were able to provide a more useful set of insights than if they had used just one or two assessment types. Barbot et al. (2019) note that particular types of creativity assessments “ … are all suboptimal, in that they incompletely represent the broader phenomenon” (p. 238). Using batteries of diverse assessments is the best path forward to a comprehensive understanding of our field’s complex construct.

Plucker and Runco noted that “Creativity researchers face one particular problem that is not always shared with other investigators of psychological phenomenon: The need to translate our work into practice” (p. 38). They noted recent application of creativity assessment strategies to a number of fields (e.g., Clements, 1991; Cole, Sugioka, & Yamagata-Lynch, 1999; Couger & Dengate, 1992; Gilbert & Prenshaw, 1996) and recommended that researchers continue to develop measures that could be easily used in applied settings. The growth of applied measurement of creativity has occurred in a wide range of fields, including athletics (Fardilha & Allen, 2020), writing (D’Souza, 2021), science education (Li, Cai, Kuznetsova, & Kurilova, 2022; Wang & Long, 2022), design (Dong, Zhu, & Li, 2021; Han, Forbes, & Schaefer, 2019), higher education (Snyder, Hammond, Grohman, & Katz-Buonincontro, 2019), and computing education (Lehmkuhl, Gresse von Wangenheim, Martins-Pacheco, Borgatto, & da Cruz Alves, 2021), among many other fields. Some of these lines of research and development are quite complex, such as Wang and Long’s application of Rasch models to the analysis of consensual assessment technique scores on science creativity tasks. This “future direction” has largely been realized.

But that is not to say that additional applied work is unnecessary, as several important topics remain to be addressed. For example, questions about the impact of rubrics on creativity have been consistently raised for years, with almost no empirical research to guide practice. Given that assessment rubrics are all the rage in K-12 and higher education – and often recommended as a best practice – the lack of research is glaring. As a case in point, the most comprehensive and thoughtful review on the use of rubrics in higher education does not mention creativity (Dawson, 2017). The goal of this research need not be a simple thumbs-up/thumbs-down on rubrics, but rather a more nuanced examination of how rubrics can be used to best support creativity (e.g., Do single-point rubrics provide the necessary balance of guidance and flexibility to optimize both student learning and creativity? See Fluckiger, 2010). Similarly, with the emphasis on group work in both education and the private sector, assessment approaches that allow for simultaneous monitoring of both individual and group creativity within groups are needed (see Stefanic & Randles, 2015).

Current state of creativity assessment

Given the mixed progress in the areas identified in 1998, what are the current issues and needed future directions of the field as its researchers work in the third decade of the 21^st century? Table 1 includes a summary of the 1998 themes and 2022 issues, advances, and potential future directions. In this section, the 2022 trends are discussed in more detail.³ The new categories overlap several of the old observations, and the lack of mention of implicit theories, for instance, should not be viewed as that work no longer being important, but rather subsumed by the broader category of continuing to improve assessment quality. More to the point, the lack of a specific category related to assessment batteries is due more to the inclusion of batteries across a number of the new categories rather than a deemphasis of their importance.

Table 1. Summary of current issues, recent advances, and future directions in creativity assessment, 1998 vs. 2022.

1998		2022
Category	Progress?	Category
Current Issues		Recent Advances
Fluency Effects	Yes	Assessment quan/qual
Generality-specificity	Conceptually	Semantic distance
Recent Advances		Current Issues
Predictive validity	Limited	Conceptions
Implicit theories	No	Quality standards
Future Directions		Predictive validity
Psychometric advances	Yes	Future Directions
Assessment batteries	No	Use of AI/ML
Applications to practice	Yes	Conceptual refinement

Recent advances

By far the biggest change since 1998 is the sharp increase in assessment quantity and quality. Every major research journal that specializes in creativity research routinely publishes assessment articles, and the 2019 special assessment issue of Psychology of Aesthetics, Creativity, and the Arts is a testament to the hundreds of studies that have been published over the past quarter century. Regarding quantity, dozens of new assessments have appeared since 2000, representing a wide range of creativity dimensions. For example, the Kaufman Domains of Creativity scales are activity checklists in the domains of everyday, scholarly, performance (writing and music), scientific, and artistic creativity (Kaufman, 2012). Not only have the K-DOC scales been the subject of several measurement studies (e.g., Kapoor, Reiter-Palmon, & Kaufman, 2021; McKay, Karwowski, & Kaufman, 2017), the scales have been used as criterion or independent variables in several subsequent studies of creativity, including studies of creative self-efficacy (Barbot, 2020), problem-solving in science (Aschauer, Haim, & Weber, 2021), and musical creativity (Tavani, Caroff, Storme, & Collange, 2016).

Similarly, the Runco Ideational Behavior Scale (Runco et al., 2001) was developed as a criterion measure for creativity studies, with the logic that attitudes toward ideational behavior were often the desired outcome of interest for many intervention studies. The scale has been examined in several measurement studies (e.g., Runco et al., 2014; Sen, 2016, 2022) and used as a criterion variable in studies of creativity (Paek, Park, Runco, & Choe, 2016; Plucker et al., 2006). Almost countless other measures have emerged this century, many with substantial psychometric support and in wide use, such as the Biographical Inventory of Creative Behaviors (BICB; Silvia et al., 2021), the Creative Achievement Questionnaire (CAQ; Carson, Peterson, & Higgins, 2005), and variations of the divergent thinking tasks mentioned earlier. The sheer quantity of new and/or improved assessments for creativity is staggering.

At the same time, the quality of these new measures is also impressive. As noted earlier, measurement studies and critical reviews have sharply increased in prevalence (e.g., Kaufman, 2019; Silvia et al., 2012), and many use modern methods such as Rasch modeling. The Consensual Assessment Technique, befitting its popularity, has also been the subject of extensive measurement studies that are helping researchers improve its use (see Cseh & Jefferies, 2019). The plethora of studies on administration and scoring of divergent thinking tasks is leading to considerable insights on how best to use this popular assessment strategy. Plucker and Makel (2010) wryly noted that users of creativity assessments needed to be patient regarding psychometric integrity, given that developers of creativity measures had not had the decades of experience and funding enjoyed by developers of achievement, ability, and intelligence tests. That patience has clearly been rewarded with a wide range of high-quality assessments.

This positive evaluation of the field’s progress comes with an important caveat: There appears to be a trend toward creating new instruments when there are already several quality instruments that are in regular use. These new instruments appear to provide incremental improvements when their authors believe them to be big leaps in new directions. The 10^th new behavior checklist will likely have less future value to the field than the development of a qualitatively different approach to assessment of mathematical creativity (Tan, Mourgues, Bolden, & Grigorenko, 2014) or significant improvements in technologies that allow traditionally hand-scored measures to be automated and scaled. Being creative about new forms of creativity assessment would serve the field and its stakeholders well.

Another important advance is the estimation of semantic distance. In brief, semantic distance is an estimate of concept dissimilarity; given that the scoring of originality in verbal DT tasks is often based on dissimilarity, the application semantic distance to that class of creativity assessments has become quite popular. In particular, this approach to analyzing DT task responses (primarily latent semantic analysis or LSA) is both theoretically grounded (Acar & Runco, 2019; Kenett, 2019) and empirically promising (Beaty & Johnson, 2021; Heinen & Johnson, 2018; Orwig, Diez, Vannini, Beaty, & Sepulcre, 2021). These methods, which decrease scoring time due to automation, are not without their critics, who note that the generalizability of LSA to other DT tasks may be limited (see Barbot et al., 2019). Regardless, these techniques may have broader applications (e.g., Guo, Ge, & Pang, 2019).

Current issues

A current issue that the 1998 analysis did not anticipate was conceptual, although it has significant implications for creativity assessment: How do we define creativity? Or more to the point, what are we assessing when we say we are assessing “creativity?” Considerable effort has been devoted to elucidating our definitions, with general agreement that originality and utility are key components (Runco & Jaeger, 2012). There is some diversity of opinion about extensions of that standard definition, with Simonton (2012) borrowing from the U.S. Patent Office definition – proposing surprising, and Plucker, Beghetto, and Dow (2004) rooting the definition within specific social contexts. Although some of the field’s leaders see these explorations as unfruitful (e.g., Silvia, 2018), these conceptual discussions have helped unify the field. To put it personally, in the early 2000s I attended a conference on creativity organized by the White House’s Office of Science Technology and Policy that was sponsored by the leading U.S. government research agencies and major private foundations. The speakers agreed on one thing: Creativity couldn’t be defined or, therefore, assessed. That was a common attitude outside the field. Today these experiences are rare, in part because the literature is replete with well-defined constructs, and researchers have disseminated their conceptual and assessment work broadly. This development is a major credit to the field, one that will benefit researchers and stakeholders for many years to come.

At the same time, two additional conceptual issues have emerged. The first deals with the broad models of creativity that the field produces. Given the wide-ranging nature of the construct, with its many personal, cognitive, and social dimensions, building robust frameworks for classifying these dimensions will be of great utility to those working with creativity assessments. As a case in point, Guilford’s (1967) original version of the Structure-of-the-Intellect model included 24 distinct types of divergent production, a number that expanded with the growth of the SOI model. Recently, Acar and Runco (2019) have proposed a new organizing framework for divergent thinking, Reiter Palmon et al. (2019) provided a detailed framework for divergent thinking assessment dimensions, and Kaufman (2019) proposed a classification of self-report measures. At this point in the field’s development – at least from an assessment perspective – further development of organizing frameworks is likely more helpful than additional debates over broad definitions, the majority of which end up being quite similar and only rhetorically distinct.

Second, sociocultural perspectives on creativity and talent development continue to gain influence within the field. These conceptualizations place a strong emphasis on the interaction among the individual and their social context (Glăveanu, 2015; Glăveanu et al., 2019; Simonton, 2019), with some scholars proposing that the individual creator is inseparable from their environment and social interactions (Plucker, McWilliams, & Guo, 2021). These perspectives have implications for creativity assessment that are just beginning to be explored and should be a focus of future creativity efforts. Measurement approaches that focus narrowly on specific aspects of creativity have less utility from this philosophical perspective, again making the use of batteries of assessments appealing in creativity research.

Another critical issue is the need for quality standards for the field’s assessments. Barbot et al. (2019) propose two broad criteria: transparency and homogenization. Transparent reporting practices are important because researchers should ensure “the particular facet(s) of the creativity phenomenon being investigated are themselves clearly defined and operationalized” (p. 238) and important but hardly new idea, both within our field (see Plucker et al., 2004) and scholarship in general. If we are not clear about what we think we are studying and how we are choosing to study it, interpretation of our research base is at best difficult and at worst nearly impossible.

Regarding homogenization, Barbot et al. call for standardization of measurement “at the subconstruct or facet level” (p. 238), meaning that specific aspects of creativity, such as DT, should have agreed-upon measures, administration protocols, and scoring strategies. Essentially, if transparency helps a research consumer understand how three studies on the same topic within creativity may differ, homogenization addresses the fact that perhaps those three research teams should be using similar methods. For example, if three groups conduct studies of the relationship between creative self-efficacy and divergent thinking within the context of math problem solving, transparency would help readers understand how each team measured and scored self-efficacy and DT differently. Homogenization would encourage the research teams to use assessment strategies that were similar if not identical, facilitating generalization of results across studies. The transparency + homogenization approach should serve as a foundation for future efforts to ensure quality and progress in a field that is, by definition, prone to divergence.

A possible, third criterion is context and inclusion. Given the growing emphasis on sociocultural perspectives in the field, assumptions about generalization across contexts – especially cultural contexts – need to be challenged. A major problem within the social sciences has been an overreliance on the use of WEIRD (Western, Educated, Industrialized, Rich, and Democratic) samples. Creativity research appears to be better than other fields at using non-WEIRD samples, which is a positive development. For example, many of the K-DOC and RIBS measurement studies have involved non-WEIRD samples (Awofala & Fatade, 2015; Kalis & Roke, 2011; Kandemir & Kaufman, 2020; Susanto, Novitasari, Rakhmat, Hidayat, & Wibowo, 2018; Tep, Maneewan, & Chuathong, 2021; Tsai, 2015), providing evidence of the impact of cultural context on these particular instruments and their resulting data. This criterion is related to transparency and homogenization but is distinct enough (and important enough) that it can stand on its own.

Although an elaborated Barbot et al. framework is a good foundation for future assessment efforts within the field, it should probably be treated as aspirational. For example, the transparency criterion was emphasized by Plucker et al. (2004), yet a follow-up study by Puryear and Lamb (2020) found only marginal improvement in reporting practices. Of course, journal editors could have a major impact on reporting practices, much as they have had a major impact on open science practices more broadly. By requiring authors of all submitted manuscripts to, at the very least, define their terms and constructs, transparency in the field’s assessment research (and research in general) would leap forward.

That said, homogenization of assessment strategies feels unlikely. Given the often heated competitiveness among researchers in the field, it is not unreasonable to question the degree to which we could collectively agree on standard approaches on even the finest-grained assessments. Given that each major research team around the world has developed their own creative behavior checklist (a small exaggeration only), is there any hope of agreement on assessment of a more complex subconstruct such as divergent thinking? Not to put too fine a point on it, but after more than a quarter century of use of the CAT, why are we no closer to knowing the optimal characteristics of raters? Again, progress toward homogenization is a desirable aspiration, but that progress will almost certainly be measured in baby steps.

A third critical issue is the need for more predictive validity studies of creativity assessment. Although the nature of validity has evolved over the past few decades (see Kane, 2016), evidence that assessment scores predict future behavior and other desirable outcomes remains important. As noted earlier, the field has made little progress in this respect and needs additional work in this area. Even short-term studies of two-to-three years would provide insights that facilitate improvement of assessment instruments, models, and strategies.

The importance of predictive validity is often debated (i.e., I have been told by more than one proponent of the CAT that predictive validity evidence is unnecessary). But our measures, especially when used in applied research, often carry the implicit promise that the data being collected is important for the future. For example, personality measures of creativity are often used in workplace hiring, and cognitive measures are used in schools. If these and other assessment strategies truly do not have any predictive power, the usefulness of creativity in general will rightfully come under scrutiny. The limited evidence is that many of the most popular measures are associated with evidence of predictive validity, providing reason for optimism. But a little more data would make the need for optimism less necessary.

Future directions

The incorporation of machine learning in the scoring of creativity assessments can certainly be classified as a “recent advance,” given the breadth and depth of ongoing efforts (see earlier discussion about semantic distance scoring). Yet the application of artificial intelligence (AI) looks to be such an important part of the field moving forward that it is best categorized as a future direction.

The use of AI in creativity assessment is particular exciting because it will allow us to do what we already do better while also allowing us to do new things. For example, the inability to scale creativity measures due to scoring burden is a major impediment to the inclusion of such measures in K-12 accountability systems and similar, wider applications in education and the private sector (Plucker & Alanazi, 2019). At the same time, unique innovations (e.g., use of semantic distance) were difficult to tackle given the required computing power and related resources. AI may help researchers address both issues by allowing us to scale things we already do well while opening doors to new possibilities that simply were not attainable in an analog world.

The field is moving assertively in this direction, with several teams around the world developing assessments (and, to a lesser extent, assessment systems) that rely on or at least use some aspects of AI (e.g., Beaty & Johnson, 2021; Dumas, Organisciak, & Doherty, 2021; Kovalkov, Paaßen, Segal, Pinkwart, & Gal, 2021; Sung, Cheng, Tseng, Chang, & Lin, 2022). These efforts predominantly focus on written language, although efforts involving figural creativity and speech are also ongoing (e.g., Cerrato et al., 2019; Cropley & Marrone, 2021; Plucker et al., in press; Shute & Rahimi, 2021). The efforts focused on written input feel like an example of the first advantage (i.e., doing what we already do better and at greater scale), and the non-written initiatives feel like the second advantage (opening ourselves up to innovative ways to assess creativity), although there is clearly overlap.

In addition, an exciting but rarely mentioned benefit of these projects is the sheer number of computer scientists, engineers, and other non-usual suspects contributing to our field. Much as the study of creativity benefited immensely from the influx of creativity researchers from East Asia and Central and Eastern Europe over the past 20 years, the growth in non-psychologists contributing to the field, in combination with continued growth of creativity research around the globe, promises to continue increasing the quantity and quality of creativity scholarship in general and creativity assessment in particular.

Although the use of modern computing power to improve creativity assessment bodes well for the future, researchers should continue to focus on conceptual issues (e.g., Basalla, Schneider, & Vom Brocke, 2022). With technology allowing us to do more and different things, greatly expanding the possibilities for creativity assessment, serious consideration of why and how we assess creativity in certain contexts will become increasingly important. Put differently, advances in the coming decade may allow us to assess creativity in almost any setting with previously unattainable speed. The possibilities are almost endless, but they are not equally valuable. What are the most promising approaches to large-scale creativity assessment, and which are the most valuable for improving human performance and quality of life?

For example, Plucker et al. (in press) have noted that the multifaceted, sociocultural, and developmental natures of creativity make it imperative to adapt assessments for different life stages and problem-solving contexts (e.g., PreK-12, postsecondary, workforce). They propose a developmental framework to serve as a guide to future development of creativity assessment systems, suggesting, for example, that domain-general assessments are appropriate for young children, domain-specific measures become more important for high school and college students, and domain- and even task-specific measures are most relevant when working with adults in the workforce. Plucker et al. also note that much of the AI-assisted work in the field focuses on cognition and process, which are an important aspect of creativity but not the only important aspect. Conceptual work on creativity assessment design and implementation may not be as sexy as developing machine language models, but it is necessary for the field to move efficiently in positive directions (see also Vartanian, 2014).

Conclusion

It is difficult to transport oneself back in time 25 years to answer the question, “If you knew what was going to happen over these next few decades, would you feel like progress was made?” Based on the analysis above, with progress in some areas but not others (and even a little backsliding), it is tempting to conclude that the authors of the 1998 paper would deliver a mixed verdict. But more likely, I suspect we would have been thrilled. The assessment of creativity has never been more dynamic, due to the tremendous volume of such work, the quality and technical sophistication of that work, and the diversity and number of researchers conducting it. In fact, as the original paper was written in the mid- to late-90s, it would have been difficult if not impossible to envision that the assessment of creativity – and the science of creativity in general – could become this robust. Much important work remains to be done, but it will be entertaining and fulfilling to see the new directions the field takes over the next quarter-century.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. The sheer volume of creativity assessment scholarship over the past 25 years makes a truly comprehensive citation of relevant work impractical – the reference list alone would be several dozen pages long. The goal of this paper is to highlight key studies and representative work and not to provide an encyclopedic bibliography.

2. This recommendation is distinct from that of Barbot et al. (2019) that researchers using DT measures use a representative range of DT tasks rather than relying only one or two such tasks in their studies, although that recommendation is well-taken.

3. The authors of the 1998 paper cannot remember why they discussed critical issues before recent advances, which makes little temporal or logical sense.