Exploring the Utility of Bayesian Truth Serum for Assessing Design Knowledge

Abstract

Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

NOTES

Notes

Funding. This work was supported in part by the National Science Foundation under award no. IIS 06-13806 and in part by the Intelligence Advanced Research Projects Activity (IARPA) via the Department of Interior National Business Center contract number D11PC20058. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions expressed herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.

Supplemental Material. Supplemental data for this article can be accessed on the publisher's website. The supplementary materials contain the full Bayesian Truth Serum Design Thinking survey.

HCI Editorial Record. First manuscript received February 15, 2013. Revisions received August 1 and August 29, 2013. Accepted by Scott Klemmer. Final manuscript received November 1, 2013. — Editor

Additional information

Notes on contributors

Scarlett R. Miller

Scarlett R. Miller ([email protected]) is an industrial engineer with an interest in design cognition, engineering design, and human–computer interaction; she is an Assistant Professor in the School of Engineering Design at The Pennsylvania State University.

Brian P. Bailey

Brian P. Bailey ([email protected]) is a computer scientist with an interest in design computing and cognition; he is an Associate Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign.

Alex Kirlik

Alex Kirlik ([email protected]) is a cognitive engineer with an interest in designing collaborative human–computer systems; he is a Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign.