Outcomes Assessment of Role-Play Scenarios for Teaching Responsible Conduct of Research

Abstract

We describe the summative assessment of role-play scenarios that we previously developed to teach central topics in the responsible conduct of research (RCR) to graduate students in science and engineering. Interviews with role-play participants, with participants in a case discussion training session, and with untrained students suggested that role-playing might promote a deeper appreciation of RCR by shifting the focus away from wanting to simply “know the rules.” We also present the results of a think-aloud case analysis study and describe the development of a behaviorally-anchored rating scale (BARS) to assess participants' case analysis performance.

Keywords:

• role-play
• assessment
• responsible conduct of research

ACKNOWLEDGMENTS

This material is based upon work supported by the National Science Foundation under Grant EEC-0628814. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Arkansas State University, the National Science Foundation, Procter & Gamble, Singapore Management University, the University of Illinois, or The University of Tulsa.

Notes

This research was initiated while all of the authors were affiliated with the University of Illinois at Urbana-Champaign.

1. We asked ninety-seven students who attended a role-play session and thirty students who attended case discussion session, all from the same department, to rate their overall experience in the session on a scale from 1 (very bad) to 5 (very good) scale. Both sessions were presented by the same individual (one of the authors). Overall, reactions to both sessions were neutral (rating of 3) or positive (rating of 4 or 5). The mean rating from the role-play participants was 3.5 (SD = 0.69) and the mean from the case-analysis discussion participants was 4.1 (SD = 0.61). A two-tailed t-test for the difference between the means was significant (t = –4.07, p < .05; Cohen's d = .92). Cohen's d is an effect size measure that is calculated by taking the mean difference divided by the pooled standard deviation; it can be interpreted as the number of standard deviations separating the score of an average member of one group from an average member of the other group. In this case, the average member of the case discussion group rated the session 0.92 standard deviations higher than the average member of the role-play group.