Effect of Test-Taking Venue and Response Format on Political Knowledge Tests

Abstract

This experiment examined the effect of testing venue (online vs. classroom) and test format (multiple choice, true/false, open-ended) on performance on a test of political knowledge. Political knowledge test scores were higher in the online open-ended condition than in the classroom open-ended condition, with no evidence of venue differences for the multiple-choice or true/false format. This pattern is consistent with self-deceptive enhancement motivations. Given the proliferation of online surveys in academic research, these data enhance our understanding of the impact of data collection procedures on the validity of measures of seminal communication constructs.

Notes

¹ Mondak’s (2001) reported response rates are high relative to our sample. This result occurred because Mondak’s questions included a “don’t know” response on the survey. He argued that including a “don’t know” response improves response rates on knowledge tests, and therefore should become standard practice for national survey organizations. Mondak takes issue with low response rates and the lack of information a nonresponse provides for public opinion research. Consequently, his work focuses on achieving higher response rates for public opinion polling. The inclusion of a “don’t know response” coupled with a nationally representative sample contributes to the discrepancy between Mondak’s response rate and the rate obtained here.

² Student major was also treated as a fixed-effect along with venue and format. The results from this ANOVA produced outcomes similar to those treating student major as a covariate.

³ A full version of all three political knowledge tests is available upon request. Please direct any inquiries to the lead author. The political knowledge scale created for this study tested political knowledge across five domains: people in politics, current events, foreign affairs, history, and knowledge of the political system. A model testing the hypothesis that all items were indicators of the same underlying factor was conducted, maximum likelihood criterion being employed to estimate the parameters. The outcome of this confirmatory factor analysis indicated that the data were consistent with the model, RMSEA=.054.

⁴ Performing these analyses on the scores without correcting for guessing produced outcomes that lead to the same substantive conclusion, save one. As Table 1 indicates, as predicted, scores were lowest in the open-ended format condition with scores in the true/false and multiple choice conditions approximately equal to one another.