A Comparison of Assessment Methods and Raters in Product Creativity

Abstract

Although previous studies have attempted to use different experiences of raters to rate product creativity by adopting the Consensus Assessment Method (CAT) approach, the validity of replacing CAT with another measurement tool has not been adequately tested. This study aimed to compare raters with different levels of experience (expert ves. nonexpert raters) using both CAT and the product creativity measurement instrument (PCMI) to assess the product creativity of 56 design works based on a design competition. The results showed that nonexpert raters who used either CAT or PCMI had higher interreliability than expert raters. Using PCMI was found to result in higher correlation than using CAT for the expert and nonexpert raters, although the correlation between the CAT and PCMI methods was statistically insignificantly different. After regression analysis, the results showed that all PCMI items had higher explanatory power for the creativity scores using CAT and, moreover, the nonexpert raters were found to have higher explanatory power than the expert raters. Based on these results, it is recommended that the use of both nonexpert raters and PCMI is an alternative way of enhancing the flexibility of product creativity assessment.

Acknowledgments

We thank the Cultural & Creative Industries Center, National Taiwan Normal University and Associate Professor Liao Wei-Ming for providing design works of the 2009 Taiwan International Student Design Competition.

Notes

Note. PCMI = product creativity measurement instrument.

Note. The dependent variable values were based on CAT creativity scores, while the independent variables were based on the nine items and the two factors of product creativity measurement instrument. VIF = variance inflation factor.

*p < .05. **p < .01.