Updating the Duplex Design for Test-Based Accountability in the Twenty-First Century

Abstract

The duplex design by Bock and Mislevy for school-based testing is revisited and evaluated as a potential platform in test-based accountability assessments today. ¹ We conclude that the model could be useful in meeting the many competing demands of today's test-based accountability assessments, although many research questions will need to be answered in future studies.

¹The term theory of accountability refers to a specific set of ideas for proposing that accountability leads to systemic improvements. Test-based accountability refers to the use of tests results to reach accountability decisions at the student or aggregate levels. The term assessment design refers to a set of principles, such as evidence-centered design (Mislevy, Steinberg &, Almond, 2003), or the specific design of an assessment or test. The duplex design discussed in this paper is an example of such a specific design.

ACKNOWLEDGMENTS

We are most grateful to Randy Bennett, Dan Eignor, Michael Kane, Dan Koretz, and Wendy Yen for valuable editorial and substantive suggestions. The comments of an anonymous reviewer were most useful in improving the manuscript.

Notes

¹The term theory of accountability refers to a specific set of ideas for proposing that accountability leads to systemic improvements. Test-based accountability refers to the use of tests results to reach accountability decisions at the student or aggregate levels. The term assessment design refers to a set of principles, such as evidence-centered design (Mislevy, Steinberg &, Almond, 2003), or the specific design of an assessment or test. The duplex design discussed in this paper is an example of such a specific design.

² http://www2.ed.gov/programs/racetothetop/index.html, see especially testimony by Randy Bennett, Barbara C. Buckley, Ed Haertel, Scott Marion, Jim Pellegrino, Gary W. Phillips, Edys S. Quellmalz, and Lorrie Shepard.

³See, for example, descriptions of the “V-model” (V-model, 2004, July 22) used in software engineering comprising these five steps.

⁴The “theory” proposed by Levin can be thought of as a generic “theory of action”. The latter term derives most directly from the work of Wholey (Bennett, 2010; Wholey, 1979), and it is used increasingly in educational accountability to present a pro forma argument for the effects of a specific approach to test-based accountability and has become a requirement in Race to the Top. The concept of theory of action also derives from the work of Argyris, although with a different sense and emphasizing the beliefs and behavior of participants in a given context. Applied to an educational context (Argyris, 1997), the sense is entirely applicable also to the present discussion.

⁵See (Carlson, undated) for a discussion of several accountability models.

⁶ http://www2.ed.gov/programs/racetothetop-assessment/competition-overview.pdf

⁷It is not the case, as suggested by a reviewer, that a two-stage adaptive design is a lesser form of adaptive testing compared to item-wise adaptive testing. Experience with several admissions tests suggests that on item-wise adaptive testing can present serious challenges. For example, controlling item exposure is difficult with such a design. A two-stage design because it consists of several forms, makes item exposure much more feasible.

⁸A competing model was proposed earlier also based on IRT but was not adaptive (Bock & Mislevy, 1981).

⁹Information about PDQ can be found at http://www.ets.org