An Alternative Decision-Making Procedure for Performance Assessments: Using the Multifaceted Rash Model to Generate Cut Estimates

Abstract

This article describes an alternative approach to setting standards for performance assessments. The procedure was designed for use in low-budget, relatively low-stakes contexts where it is not possible to bring expert judges together. The procedure that allowed participant judges to work individually throughout the process was an effort to compensate for the lack of face-to-face interactions between judges. The viability of the proposed procedure in generating workable cut estimates was examined via its application to a certification examination in medical translation. Data were collected from a case study conducted by Kozaki (2004) and its replication, which were linked together by the Multifaceted Rasch Model into a single frame of reference. The results showed that the judges' intentions were overall well represented in the generated cut estimates and that the decision making was realistic. It is suggested that the proposed procedure should offer a viable alternative in relatively low-stakes environments to full-fledged standard-setting procedures and that thus it may be of benefit to examiners who need principled approaches to setting cut estimates for performance assessments with constrained budgets.

ACKNOWLEDGMENTS

This article was developed from a portion of my EdD dissertation at Temple University Japan. My thanks go to Steven Ross (Chair), David Beglar, Judy Noguchi, Miyuki Sasaki, and Kenneth Schaefer, who served on my dissertation committee. I would also like to thank three anonymous LAQ reviewers for their valuable comments and constructive feedback.

Notes

¹This objective is based on the notion of “move-structure” in teaching English for specific purposes, in which the role of patterns of discourses is emphasized in reading and writing specialized texts. The move-structure is composed of “moves” and “steps.” Although a series of moves is thought to accomplish the goal of the overall discourse, steps under each move function to realize the purpose of the move. It is through an adequately sequenced series of moves and steps that communicative purpose is believed to be achieved among members of the discourse community. The notion schema convention here refers to conventionalized expressions used as linguistic signals that help notify the reader of particular moves and steps.

²Choice of some types of verb form (e.g., reporting verbs) and the tense plays the significant role when authors attempt to create their research spaces (e.g., by denoting their stance toward cited work) and when they set the level of claim and the degree of generalization for their own work. These features thus require special attention in research articles and deserve to be highlighted in a curriculum for medical translation.

³These boundaries are admittedly arbitrary in the sense that some other number, say, 67% (two thirds of the judges) as opposed to 75% (three fourths of the judges), could have been used, if more judges had participated in each session. Note here that as the number of judges (and examinees) increases, more elaborate, fine-tuned ground rules can be established, which would make it possible to identify more precise boundaries of the potential area of decision making.

⁴I acknowledge that with this small-size data, conclusive answers cannot be given to the research questions. The purpose of this article, however, is to advance the potential of the proposed alternative procedure in generating workable cut estimates and show how, with the aid of the Multifaceted Rasch Model, panels' judgments can be studied and dealt with.