Revisiting Individual Creativity Assessment: Triangulation in Subjective and Objective Assessment Methods

Abstract

Compared to the significant development of creativity studies, individual creativity research has not reached a meaningful consensus regarding the most valid and reliable method for assessing individual creativity. This study revisited 2 of the most popular methods for assessing individual creativity: subjective and objective methods. This study analyzed 1,500 individuals to investigate whether the methods for assessing individual creativity affect the measurement outcomes of individual creativity. Findings indicated that subjective assessments have a smaller variance a higher mean and a moderate but significant correlation with objective assessment methods. Such differences can be motivated by social desirability, consistency motif, illusory superiority, and leniency biases. Based on these findings, this study highlighted the need to acknowledge how subjective and objective assessment methods may affect individual creativity assessment outcomes.

The assessment of creativity is one of the central topics in individual creativity research (e.g., Kaufman, Plucker, & Baer, 2008; Puccio & Murdock, 1999). The growth of creativity studies led to the development of many methods for assessing creativity, which explored diverse aspects of individual creativity from personal to various contextual factors. Despite significant progress over the last few decades, there still exists an ongoing debate on the reliability and validity of creativity assessment methods (e.g., Cropley, 2000; Hocevar, 1981; Runco, 1986; Wallach & Kogan, 1965). As Plucker and Makel (2010) acknowledged, creativity assessment is among the topics that incite the most discourse in creativity studies.

Individual creativity assessment methods are often classified as subjective and objective, depending on their characteristics and measurement tools. Many previous studies on individual creativity used either subjective or objective methods interchangeably without clearly distinguishing the possible differences in measurement outcomes. Although both subjective and objective methods have their own upsides respectively, previous studies suggested that the special attention to measurement and methodological issues is still important for advancing research in theoretical relationships between constructs (Venkatraman & Grant, 1986; Venkatraman, 1989). Specifically, they suggested developing a method that simultaneously integrates the merits of objective and subjective methods. Regarding subjective assessment, they advised combining objective methods to correct for possible bias and derive a clearer picture of individual creativity (Madjar, Oldham, & Pratt, 2002; Shin, Kim, Lee, & Bian, 2012). Regarding objective assessment, they argued that it may fall short of fully capturing the dynamic interplay of the various factors of creativity (Elsbach & Kramer, 2003). They further criticized the fact that most standardized tests do not pay sufficient attention to important dimensions such as out-of-the-box thinking, originality, and flair (Elsbach & Kramer, 2003; Reich, 2001). The complex and multidimensional characteristic of creativity requires a more comprehensive approach in assessing creativity (Cropley, 2000; Feldhusen & Goh, 1995; Plucker & Makel, 2010).

Therefore, this study aimed to investigate how different assessment methods affect measurement outcomes of individual creativity. Specifically, it examined whether subjective and objective assessment methods may lead to the same measurement outcomes or not. If not, it further explored how these two are different. To test the research questions, a total of 1,500 individual data in South Korea was collected. Both subjective and objective assessment methods were used to assess each individual’s creativity. Findings indicated that that subjective and objective methods showed significant differences in their assessment results. Specifically, the subjective assessment method showed a higher mean and smaller variance than the objective method. This study also found that the correlation between the two assessment methods was significant, but not as high as expected. Moderate correlations between subjective and objective assessment could possibly weaken the robustness of research conclusions. Overall, this study showed that the quality of assessment methods used for empirical research may play a crucial role in linking theory to empirical tests.

TRIANGULATION OF SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS FOR INDIVIDUAL CREATIVITY

The attempts to develop better assessment methods took many forms, such as improving perceived weaknesses of existing measures or encompassing new dimensions believed to affect creativity (e.g., Plucker & Makel, 2010; Plucker & Runco, 1998; Schoenfeldt & Jansen, 1997). Previous studies continuously acknowledged the need for methodological triangulation (e.g., Madjar et al., 2002; Perry-Smith, 2006; Runco, 1986; Shalley, Gilson, & Blum, 2000). Triangulation refers to the combination of multiple methodologies in the study of the same phenomenon for the purpose of better research design (Denzin, 1970; Mathison, 1988). A possible triangulation approach for individual creativity assessment may concern dealing with the characteristics of assessment methods, specifically subjective or objective assessment methods. A well-taken distinction criteria between objective and subjective creativity assessment methods concerns how creativity is measured and by whom it is measured. It is also possible that the distinction between objective and subjective assessment is two opposite ends of a continuum, rather than two completely distinct methods. Thus, in this study, objective and subjective methods refer to a method that has more or less subjective/objective qualities than other methods. Subjective assessment methods are more qualitative and involve analyzing data from direct fieldwork observations, surveys, interviews and written documents. On the other hand, objective methods involve more quantitative analyses that concern systematic empirical investigations through statistical, mathematical or numerical approaches (Patton, 1990). Developing objective creativity assessment methods often requires significant investment of resources, such as the long process of designing assessment structures, developing grading systems, training research administrators and sometimes developing computer algorithms. To review the upsides and downsides of each method, this study first selected the three most representative and well-taken assessment methods that have typically either subjective or objective characteristics. Then, each assessment method was individually examined in greater depth and discussion about key issues was followed.

Self-Report

Previous studies argued that individuals are most likely to be aware of subtle tasks and the occurrence and frequencies of certain creative behaviors (Janssen, 2000; Ng & Feldman, 2012). The self-reporting method directly asks the subjects to self-judge their own creativity-related abilities, perceptions or creative performance. Self-reports usually rely on relatively simple questionnaire-type methods that are fairly easy to administer. As Janssen (2000) argued, some aspects of the creative person, such as perceptions, beliefs, self-concepts, and past behaviors may not be known to peers or supervisors unless one simultaneously engages in impression management. Thus, self-reports are sometimes the only available and reliable assessment method for assessing certain aspects of creativity (Amabile, 1996; Kaufman & Baer, 2004). Well-regarded assessment methods using self-reports include Gough’s (1979) Creative Personality Scale (CPS), Amabile’s (1996) KEYS (Assessing the Climate for Creativity) and Hocevar’s (1979) and Kirschenbaum’s (1989) Creative Behavior Inventory (CBI).

On the subjective/objective assessment scale, self-reports are closest to being the most subjective of methods as they rely solely on subjective judgments of oneself. Numerous studies showed significant correlation between self-reports and other measures of creativity, such as divergent thinking (Barron, 1955; Batey & Furnham, 2006; Carson, Peterson, & Higgins, 2005; Gough, 1979). For example, Axtell et al. (2000) found that the correlation between the creativity indexes measured by self-reports and supervisor evaluations was significant and as high as 0.62. However, the self-report method has not been able to avoid criticisms, such as providing at best ‘soft data’ and being prone to misinterpretation (Podsakoff & Organ, 1986). Some studies suggested that self-reports are subject to dangers of bias due to social desirability and consistency motif (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003).

Supervisor Evaluation

Previous studies recognized that supervisors are in good position to recognize creativity in certain domains. Domain specificity, an important reason for using supervisor evaluation, recognizes that supervisors are among those most familiar with the domain and possess the expertise to judge (Hocevar, 1981). Supervisor evaluation has often been used as a common assessment tool for field studies at organizational settings (George & Zhou, 2002; Oldham & Cummings, 1996; Scott & Bruce, 1994). For example, Oldham and Cummings (1996) developed a 3-item to assess creative performance and asked questions such as “How original and practical it this person’s work?” (p. 634) on a 7-point Likert-type scale. Similarly, Zhou and George (2001) developed a 13-item scale, and asked questions such as “Is this person a good source of creative ideas?” or “Does this person come up with creative solutions to problems?” (p. 696) Zhou and Shalley (2003) noted that although these tests share many similarities, they are at the same time different, emphasizing different aspects such as innovation, creative outcomes, creative processes, and so forth.

On the subjective/objective assessment scale, while more objective than self-reports, supervisor evaluation still has subjective qualities, as it relies on supervisors’ subjective evaluations. Supervisor evaluations are also relatively simple and easy to administer. Moreover, the quantitative nature of supervisor evaluations allows for broad evaluation of the employee’s creativity due to domain specific knowledge of the supervisor, providing rich and comprehensive descriptions (Zhou & Shalley, 2003). Supervisors’ ratings have been found to significantly correlate with archival measures (Scott & Bruce, 1994). However, supervisor evaluation is not free from potential bias, such as demographic characteristics, supervisory liking, halo effects, systematic bias and social desirability (Gilson, Mathieu, Shalley, & Ruddy, 2005; Landy & Farr, 1980; Madjar et al., 2002). Some researchers criticized this method on the grounds that supervisors are not best suited to rate complex creative processes or attributes which is comparatively less obvious than objective performance that is often more obvious and visible (Janssen, 2000; Organ & Konovsky, 1989).

EXPERT EVALUATION

Experts refers to those who have specialized knowledge in a certain domain and at the same time do not have a supervisor–employee relationship with the examinee. For example, experts may include panels of experts in the relevant domain, such as judges or reviewers and professionals with advanced degrees (Amabile 1983, 1996; MacKinnon, 1962). Creative outputs such as patent disclosures, awards, or research papers all involve evaluations of various experts (Amabile, 1996; Oldham & Cummings, 1996; Scott & Bruce, 1994; Tierney, Farmer, & Graen, 1999). Experts can also include trained experts who grade standardized creativity tests such as the Torrance Test of Creative Thinking (TTCT Torrance, 1974). These experts follow strict standardized guidelines to ensure objective assessment of creativity.

On the subjective/objective scale, expert evaluation is the most objective out of the three. Expert evaluation usually relies on quantitative indexes, such as the number of patents or scores of a divergent thinking test, which can ensure greater objectivity, allows easier comparison across different examinees, and be a solution to the bias problems of more subjective assessment methods (e.g., Madjar et al., 2002; Perry-Smith, 2006; Zhou & Shalley, 2003). In particular, standardized tests are recognized as one of the most adept in assessing the cognitive processes of creativity (e.g., Guilford, 1965; Plucker & Runco, 1998). Divergent thinking frequently assessed through standardized tests is one of the single most extensively studied topics in creativity studies (Runco, 2010). However, expert evaluations have not been able to avoid several critical weaknesses. First, scholars raised questions about the definition of experts: “Who can rightfully be considered experts of the domain?” (Kaufman & Baer, 2012, pg. 84). Moreover, some objective creativity indicators such as patents and awards may only be relevant indicators for limited areas and industries (Zhou & Shalley, 2003). In spite of considerable evidence of interrater reliability, some studies argued that high interrater reliability alone is not a sufficient condition to ensure assessment validity (Kaufman & Baer, 2012).

Despite different approaches for assessing creativity, current creativity studies do not seem to apply a clear-cut distinction between subjective and objective assessment of creativity, using the two different methods rather interchangeably. This study questioned this assumption of interchangeability and developed hypotheses addressing the reasons why subjective or objective assessment methods could generate different outcomes. When assessing individual creativity, subjective assessment methods may be inherently exposed to at least two critical biases, social desirability and consistency motif (also referred to as cognitive consistency) (Podsakoff & Organ, 1986). Social desirability bias refers to the measurement error attributed to the individual tendencies in over-reporting socially desirable personal characteristics and under-reporting socially undesirable characteristics (Arnold & Feldman, 1981; Taylor, 1961; Thomas & Kilmann, 1975). This bias often prompts responses that represent the person in a more socially favorable light. For example, in a study on self-confidence and creativity, Goldsmith and Matherly (1988) found possibility of systematic bias in creativity and self-confidence scales among male students. They explained that, compared to women, men were inclined to represent themselves in what they thought was a socially desirable, masculine manner. Consistency motif describes the tendency to maintain consistency between cognition and attitudes (Heider, 1958; Osgood & Tannenbaum, 1955). Respondents participating in evaluations often have the desire to make themselves appear consistent and rational. They tend to search for similarities between questions asked and try to produce relationships that would otherwise not exist in real-life contexts (Podsakoff et al., 2003). Coupled with social desirability, respondents may prefer to maintain a consistent line in a series of answers that they think are socially favorable. This will become more prominent when they believe that socially mediated rewards (or punishments) are significant (Fisher, 1993).

Hence, due to these tendencies, compared to objective assessment outcomes, subjective assessment outcomes of individual creativity will likely have a frequency distribution that has a higher frequency of values near the mean, concentrated within a certain ‘preferred’ range of scores. Thus:

Hypothesis 1:

The variance of subjective assessment measures on individual creativity will be smaller than the variance of objective assessment measures on individual creativity.

Compared to objective assessment methods, the respondents to subjective assessment methods may have the tendency to overestimate themselves due to biases of illusory superiority and leniency. The concept of illusory superiority, also identified as a sense of relative superiority (Headey & Wearing, 1988) or the above average effect (Dunning, Meyerowitz, & Holzberg, 1989), describes the tendency to treat all members of a group, or to describe oneself as above average (Brown, 1986). Leniency bias refers to the tendency of evaluators rating themselves as higher than they should (Guilford, 1954). Leniency bias may be a more serious problem for certain raters, such as those who have the tendency to give overall higher marks as compared to other judges or those that give higher marks to certain examinees, such as those with whom they are ego involved. For example, in the case of peer and supervisor evaluations, Moneta, Amabile, Schatzel, and Kramer (2010) found that evaluators with a high degree of agreeableness tended to generally rate examinees as more creative than other evaluators. In addition, the aforementioned social desirability could also prompt respondents to conform to certain social expectations, such as giving higher scores for more socially desired items. Compared to objective assessments, subjective assessment methods are more prone to bias, because most evaluations are administered in a social context where examinees may be concerned about their self-evaluation results. Likewise, evaluators may also be concerned about the possibility that their personal relationships might interfere with their evaluation results. In comparison to objective assessments, the three tendencies mentioned previously may lead evaluators to give higher ratings, resulting to an upwardly biased evaluation. Thus:

Hypothesis 2:

The mean of subjective assessment measures of individual creativity will be higher than the mean of objective assessment measures of individual creativity.

As predicted, if both the mean and variance of subjective and objective assessments are significantly different, a critical question is whether subjective and objective assessment methods effectively measure the same construct. To ensure triangulation between subjective and objective assessment methods, it is necessary to make sure that there exists convergent validity or strong correlation, showing agreement between two different methods in measuring a single construct (Campbell & Fiske, 1959). If multiple and independent assessment methods do not reach the same conclusions, it is evidence of weak convergent validity (Jick, 1979). Very weak convergent validity could possibly threaten the entire validity of research results (Campbell & Fiske, 1959). Thus, checking for convergent validity is an important step to determining the usefulness and effectiveness of different measurement methods in the research design process (Amabile, Conti, Coon, Lazenby, & Herron, 1996; Hocevar, 1981).

Whether subjective and objective assessments of individual creativity reach the same conclusions has been an on-going debate. For example, there exists evidence of significant correlations between various subjective assessment methods and between subjective and objective methods (Janssen, 2000; Scott & Bruce, 1994). Others however, found significant differences between several assessment methods on individual creativity. For example, Oldham and Cummings (1996) found that subjective assessments, such as supervisor ratings, were more closely aligned with objective measures, such as number of patents, but not with measures of creative process. Furthermore, Moneta et al. (2010) found that there was a significant discrepancy even within subjective assessment methods. They showed that peer, supervisor, and self-evaluations overlap to only to some extent, and each method simultaneously addressed different sets of cues of creative performance. These confounding results of previous studies clearly suggest that it is still an important empirical question to investigate the convergent validity issue between subjective and objective assessments of individual creativity. Thus:

Hypothesis 3:

There will be a significant positive correlation between objective and subjective assessment measures of individual creativity.

METHOD

Participants

The participants in this study were 1,500 Koreans who voluntarily took part in this research while they were taking classes on creativity and innovation. Demographic information (age, gender, education level, and current occupation) was collected prior to the test. The ages of the participants ranged from 16 to 58 (M = 34.9, SD = 12.7), they were either students (middle school, high school, university and graduate) or MBA students who had previous work experience in various industries such as finance, automobile, electronics, construction, pharmaceutical and public service. There were 1,333 men (88%) and 167 women (12%) in the sample. All participants gave informed consents prior to their participation and were debriefed regarding the purpose of the study.

Procedure and Materials for Individual Creativity Assessment

The test was administered in 2-hr group testing sessions over a total period of 8 years. Different test sets were developed for this study, one for subjective and another for objective creativity assessment. The subjective assessment score was measured through self-ratings by participants. Self-ratings followed and reintegrated previous subjective measurements (e.g., Gough’s CPS, 1979; Oldham & Cummings, 1996; Tierney et al., 1999; Zhou & George, 2001), which mainly asked participants to self-rate their creative interests, attitudes, and self-perceptions of their own creative ability using a 7-point Likert-type scale ranging from 1 = strongly disagree to 7 = strongly agree. Some samples of frequently used subjective indicators include, “Suggests new ways to achieve goals or objectives.” “Not afraid to take risks” (Zhou & George, 2001, p. 696) and “Tries new ideas or methods first” (Tierney et al., 1999, p. 620). Questions developed for this section are quite similar to the aforementioned subjective assessment measures of creativity. It consisted of 45 items and participants were given 10 min. These questions were designed to evaluate dimensions of self-assessed creativity such as boldness, motivation, persistence, acceptability, and curiosity. The subjective assessment score for each participant was calculated as a single index that ranged from 0 to 100.

Objective assessment followed previous studies that focus on examining creative cognitive skills (Guilford, 1967a, 1967b; Torrance, 1962, 1974). It was designed to measure creative thinking abilities, more specifically divergent and convergent thinking abilities. Previous studies suggested that divergent thinking is a good predictor of creative potential and achievement (e.g., Basadur, Graen, & Scandura, 1986; Guilford, 1967a; Torrance, 1962; Woodman, Sawyer, & Griffin, 1993). Divergent thinking requires the ability to make unique combinations of ideas, make remote associations, and transform ideas into unusual forms (Cropley, 2006; Mednick, 1962; Wallach & Kogan, 1965). To objectively assess divergent thinking ability, participants in this study were given 30 short-essay questions to complete within 50 min, with an average time of 100 sec for each question. Participants were asked to produce multiple or alternative answers from given information or situations that were described in verbal or figural form.

Previous research also highlights convergent thinking ability as a critical determinant of individual creativity (Cropley & Cropley, 2012; Cropley, 2006; Runco, 2004). Convergent thinking is the thinking process that involves logical search and information processing to derive a single best answer to a predefined question. Although divergent thinking alone would often just lead to wild ideas that might be creative but unrealistic, convergent thinking is important in making sure that ideas are useful and relevant (Runco & Acar, 2012). Lonergan, Scott, and Mumford (2004) also suggested a two-step process that combines novel idea generation (divergent thinking) and simultaneous exploration of its effectiveness (convergent thinking). Following the previous arguments, divergent and convergent thinking ability scores were given equal weight to calculate the objective assessment score. Convergent thinking ability was assessed through 15 multichoice items completed within 30 min with an average time of 120 sec per question. Participants were mainly asked to solve questions with a single answer. This required qualities such as speed, accuracy, and logic. Overall, the objective creativity score consisted of the sum of divergent and convergent thinking scores. Objective assessment questions in this study were designed to evaluate several subcomponents of creative thinking ability, such as novelty, synthesis, elaboration, complexity, redefinition, fluency, and flexibility. It was calculated as a single index that ranged from 0 to 100.

Coding and Scoring

Various types of questions, such as short essays, multiple choice, and open-end questions, were used to assess individual creativity. Relying on the coding manual developed by the authors, two research assistants coded all answers from participants for the analysis. Cronbach α for the interrater reliabilities across items and participants ranged from .87 to .96. Once all coding procedures were completed, computer algorithms automatically calculated both subjective and objective assessment scores. Computer algorithms were developed in collaboration with computer program engineers from a private research institute, the iCreate Creativity Institute. They were refined and adjusted through multiple pilot tests and grading exercises for over 2 years. Sample test questions are available from the authors upon request.

RESULTS

Descriptive Statistics

Table 1 displays descriptive statistics of all the variables. Participants’ subjective creativity score mean (M = 72.8, SD = 6.5) was significantly higher than that of their objective creativity score (M = 44.1, SD = 22.9). The score range for objective creativity (range: 0.3–99) was much wider than that of subjective creativity (range: 44.7–100). Figure 1 shows the frequency distribution graph for objective and subjective creativity scores. As shown in Figure 1, the subjective creativity score shows a higher mean with lower variability. Skewness and kurtosis values for objective and subjective creativity scores were examined to check the normality of each variable’s distribution. Skewness and kurtosis for both scores were within acceptable limits.

Table 1 Descriptive statistics

Variables	N	Min	Max	M	SD
(1) Subjective Creativity Score	1,500	44.7	100	72.8	6.5
(2) Objective Creativity Score	1,500	0.3	99	44.1	22.9
(3) Objective Creativity Score: Convergent Ability	1,500	0	100	43.6	27.7
(4) Objective Creativity Score: Divergent Ability	1,500	0	100	44.7	28.4
(5) Age	1,500	16	58	34.9	12.7
(6) Gender	1,500	0	1	0.9	0.3
(7) Education Level	1,500	1	4	2.7	1.0

Note: Gender (1 = Male, 0 = Female), Education level: 1- Secondary school student/ High school diploma, 2- Undergraduate/ College student, 3- Undergraduate/ College diploma, 4- Graduate student/ Graduate diploma (MA, PhD).

Figure 1 Frequency distributions of objective and subjective assessment of individual creativity.

Objective and Subjective Creativity Score Analysis

A variance ratio test was conducted to test hypothesis 1. The test for is given by , which is distributed as F with (n–1, n–1) degrees of freedom. Results of the F-test showed a statistically significant difference between these two variances, F(1499, 1499) = 12.59, p < .001. The variance ratio of the objective creativity score to the subjective creativity score was much greater than 1, indicating greater variability for the objective creativity score. Overall, the findings supported hypothesis 1. To test the second hypothesis, paired t-tests were conducted. Paired t-tests are often used to test the statistically significant mean difference of creativity measurement scores (e.g., Runco, 1986). The results are shown in Table 2. They indicate that the mean difference between objective and subjective creativity scores is statistically significant, t(1499) = 49.38, p <.001. The study found that the mean of the subjective creativity score (M = 72.8) was significantly higher than that of the objective creativity score (M = 44.1). Overall, Hypothesis 2 was also supported. As shown in these results, participants had the tendency to rate themselves as having moderate levels of creative abilities, a quality which is generally socially favorable. The majority of participants showed similar patterns, exposing the subjective creativity score to greater systematic overestimation bias than the objective creativity score (Goldsmith & Matherly, 1988).

Table 2 Mean differences and paired t-test for subjective and objective creativity scores

Pairs	M diff. (1)-(2)	SE M diff. (1)-(2)	t	df	p
(1) Subjective Creativity Score
(2) Objective Creativity Score	28.7	0.581	49.38	1,499	0.000

n = 1,500.

Note: Two-tailed t-test.

Correlation Analysis

The relevant bivariate correlations are reported in Table 3. Supporting hypothesis 3, the subjective creativity score was significantly correlated with the objective creativity score (r = .20, p <.01). The moderate level of correlation between the two scores fulfills the necessary condition for the triangulation of creativity measurement. Among other variables, correlations between the objective creativity score and both the convergent (r = .81, p < .01) and divergent thinking (r = .82, p < .01) ability scores were significantly correlated. Considering that the objective creativity score is the sum of the convergent and divergent thinking ability scores, a high level of positive correlations among these variables is naturally expected. Convergent and divergent thinking ability is found to be significantly correlated (r = .33, p < .01) at a moderate level. These results suggest that individuals with high objective creativity scores are likely to have extremely high scores in either convergent or divergent thinking or ability.

Table 3 Correlation

	(1)	(2)	(3)	(4)	(5)	(6)	(7)
(1) Subjective Creativity Score	1
(2) Objective Creativity Score	0.20**	1
(3) Convergent Ability	0.14**	0.81**	1
(4) Divergent Ability	0.18**	0.82**	0.33**	1
(5) Age	−0.02	−0.35**	−0.31**	−0.25**	1
(6) Gender	−0.00	−0.15**	−0.08**	−0.17**	0.20**	1
(7) Education Level	0.11**	−0.11**	−0.16**	−0.03	0.70**	0.00	1

Note: n = 1,500.

**p < 0.01; *p < 0.05 (two-tailed tests).

Gender (1 = Male, 0 = Female), Education level: 1- Secondary school student/ High school diploma, 2- Undergraduate/ College student, 3- Undergraduate/ College diploma, 4- Graduate student/ Graduate diploma (MA, PhD).

DISCUSSION

The significance of individual creativity assessment led to the development of numerous measurement methods over the last few decades (Kaufman et al., 2008; Puccio & Murdock, 1999). Although progress has been significant, such diversity of assessment methods itself has illuminated the need to investigate ways to improve the reliability and validity of individual creativity assessment. Feldhusen and Goh (1995) acknowledged that scholars need to combine different measures of creativity to derive a comprehensive picture of creativity in individuals. To address this issue, this study aimed to investigate how subjective and objective assessment methods differently affect measurement outcomes of individual creativity. Analyzing 1,500 individuals, it showed that subjective assessment results have a smaller variance than objective assessment results (SD = 6.5 and SD = 22.9, respectively). This study argues that such results can be attributed to social desirability and consistency motif. This study showed that subjective assessment results tend to be more upwardly biased with a higher mean compared to objective assessment (M = 72.8 and M = 44.1 for subjective and objective assessment, respectively). These differences may be associated with illusory superiority, leniency biases and social desirability that are rooted in social settings.

Furthermore, the correlation between the two assessment methods is significantly positive and greater than zero (r = .20, p < .01). Social science research has not specified statistical threshold levels of correlations sufficient to confirm convergent validity (Carlson & Herdman, 2012). For example, Campbell and Fiske (1959) reviewed 12 studies and reported 144 estimates of convergent validities ranging from r = .20 to .82. Other researchers (Wall et al., 2004) investigated studies with correlations between subjective and objective measures of organization performance and found associations ranging from r = .26 to r = .65. The correlation coefficient of r = .20 in this study, although positive and significant, is rather weak in comparison to the ranges of correlations reported in other studies. Thus, this study may not deny that there are no critical differences between the two assessment methods. These results advise scholars to be very cautious when using the two methods interchangeably. With this in mind, it may be a more reasonable approach to combine multiple assessment methods for individual creativity assessment. This study could provide solutions to an ongoing measurement issue in creativity studies. Many empirical studies on individual creativity that use various assessment methods often relied on a single assessment method only and admitted to measurement issues only as limitations or for future research (e.g., Perry-Smith, 2006; Shalley et al., 2000; Tierney & Farmer, 2002). Nonetheless, persistent calls for the need to incorporate different or multiple methods to enhance the robustness of research results has not ceased (e.g., Farmer, Tierney, & Kung-Mcintyre, 2003; Oldham & Cummings, 1996). Campbell and Fiske (1959) suggested triangulation strategies that use complementary and multiple methods to enhance the validity of analysis results. Jick (1979) acknowledged that each method has its own strengths and weaknesses, proposing that triangulation helps exploit strengths and neutralize weaknesses.

This study could also provide a foundation to better understand conflicting results of previous empirical studies. Considering the moderately low levels of significant correlation between subjective and objective assessment outcomes, measurement-related issues as a possible reason for mixed findings cannot be eliminated. For example, some studies showed intrinsic motivation as critical to creativity (e.g., Amabile, 1979, 1985; Amabile, Hill, Hennessey, & Tighe, 1994); others reported that the relationship was, at best, weak and/or statistically nonsignificant (e.g., Dewett, 2007; Perry-Smith, 2006; Shalley et al., 2000). In particular, Grant and Berry (2011) specifically raised methodological issues about intrinsic motivation. They argued that intrinsic motivation is more consistently associated with creativity scores measured by self-reports, rather than those assessed by either observer ratings or archival measures. Mixed findings are present in many other studies, such as the relationship between individual creativity and transformational leadership (Jaussi & Dionne, 2003; Shin & Zhou, 2003), organizational learning (Gong, Huang, & Farh, 2009; Redmond, Mumford, & Teach, 1993), and intrinsic and extrinsic rewards (Eisenberger & Rhoades, 2001; Hackman & Oldham, 1980). Addressing these conflicting findings from a measurement perspective could refine and enhance future creativity research.

This research is also not free from limitations that could provide meaningful opportunities for future research. First, self-reports and expert evaluation with computer algorithms were utilized to compare subjective and objective assessment methods. Although encompassing all existing methods may not be possible, future studies could incorporate other and multiple methods, such as archival, historiometric studies, and experimental measures. By examining how each method affects and/or changes analysis results, future research could provide further fruitful discussions. Second, this study mainly examined cognitive capabilities that consist of convergent and divergent thinking ability. This could be further divided into multiple subcomponents or multiple dimensions, such as looking at divergent thinking form a product, or process perspective. Future studies could refine findings by answering how the assessment methods of individual creativity could affect such subcomponents or substructures of creativity. Third, future studies could integrate other conditional factors that could possibly affect individual creativity. For example, they could address possible associations between assessment methods and social and/or hierarchical contexts. Last, subjective and objective assessment methods and their triangulation could also be relevant and important for team or organizational creativity. Future research could examine measurement issues at different levels of analysis in creativity studies.

This study extends existing creativity literature by revisiting assessment method issues and individual creativity. Measurement issues, if serious, can significantly change either magnitudes or directions of research findings. In research design for creativity studies, our study argues that it lacked a thorough and in-depth consideration regarding the possible triangulation of objective and subjective assessment methods.