Unpacking Response Process Issues Encountered When Developing a Mathematics Teachers’ Pedagogical Content Knowledge (PCK) Assessment

ABSTRACT

It is essential for items in assessments of mathematics’ teacher knowledge to evoke the desired response processes – to be interpreted and responded to by teachers as intended by item developers. In this study, we sought to unpack evidence that middle school mathematics teachers were not consistently interacting as intended with constructed response (i.e. open-ended) items designed to assess their pedagogical content knowledge (PCK). We analyzed recent data derived from think-aloud interviews with 13 teachers involving 38 assessment items designed to tap PCK regarding proportional reasoning. Five key issues associated with undesired response processes were identified: (1) scenarios provided insufficient information, (2) content knowledge (CK) and PCK elements were confounded, (3) questions asked about the scenarios lacked specificity, (4) items contained distracting text and/or visual elements, and (5) differences between math education research and classroom teacher work cultures led to unanticipated interpretations of items. These issues were associated with teacher responses that were problematic (e.g. vague, off topic, etc.). In addition, we suggest that obtaining response process evidence is critical, and the way it is obtained may impact the average difficulty of the final pool of assessment items developed.

KEYWORDS:

• Assessment item development
• construct-irrelevant responses
• constructed response
• pedagogical content knowledge
• response process validity evidence

Acknowledgments

The researchers would like to acknowledge Yasemin Copur-Gencturk for her support with this research.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The work was supported by the National Science Foundation under Grant Number 1813760.