What Cognitive Interviewing Reveals about a New Measure of Undergraduate Biology Reasoning

Abstract

Reasoning skills have been clearly related to achievement in introductory undergraduate biology, a course with a high failure rate that may contribute to dropout of undergraduate STEM majors. Existing measures are focused on the experimental method, such as generating hypotheses, choosing a research method, how to control variables other than those manipulated in an experiment, analyzing data (e.g., naming independent and dependent variables), and drawing conclusions from results. We developed a new measure called inference making and reasoning in biology (IMRB) that tests deductive reasoning in biology outside of the context of the experimental method, using not previously taught biology content. We present results from coded cognitive interviews with 86 undergraduate biology students completing the IMRB, using within-subjects comparisons of verbalizations when questions are answered correctly versus incorrectly. Results suggest that the IMRB taps local and global inferences but not knowledge acquired before study or elaborative inferences that require such knowledge. For the most part, reading comprehension/study strategies do not help examinees answer IMRB questions correctly, except for recalling information learned earlier in the measure, summarizing, paraphrasing, skimming, and noting text structure. Likewise, test-taking strategies do not help examinees answer IMRB questions correctly, except for noting that a passage had not mentioned specific information. Similarly, vocabulary did not help examinees answer IMRB questions correctly. With regard to metacognitive monitoring, when questions were answered incorrectly, examinees more often noted a lack of understanding. Thus, we present strong validity evidence for the IMRB, which is available to STEM researchers and measurement experts.

Keywords:

• Biology
• cognitive interviews
• reasoning

Notes

1 For clarification, the term inference is used for two purposes in this study. The first is as a measured construct for the IMRB. The second usage stems from using a validation argument approach to supporting the use of the IMRB for its intended purposes. Within a validity argument for an assessment, inferences are generated that have associated assumptions. Those assumptions are then supported through the evaluation of research study results, existing literature, and reasoned semantic arguments.

2 Description inference: The IMRB consists of clearly defined and developed measurement targets.

3 Explanation inference: The expected score on the IMRB can be used to make classification decisions such as identifying examinees who are “at risk” of dropping out of STEM-related fields of study.

4 Extrapolation inference: The classification of “at risk” can be interpreted to mean that the examinee lacks an appropriate level of inference-making skill for their STEM-related coursework.

5 Evaluation inference: Observed performance on the IMRB is adequately transformed into a test score that can be used to represent the examinee’s ability to make inferences.

6 Rereading the question could have been placed with low-level strategies, as we do not believe it would produce construct-irrelevant variance. However, it was one of the new test-related codes added to an existing coding scheme that has been previously used with connected text only. We therefore place it in the test-taking strategies category.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A160335 to the University of Illinois, Urbana-Champaign. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.