Are Online and Paper Tests Comparable? Evidence from Statewide K-12 Tests

ABSTRACT

We investigate two research questions using a recent statewide transition from paper to computer-based testing: first, the extent to which test mode effects found in prior studies can be eliminated; and second, the degree to which online and paper assessments offer different information about underlying student ability. We first find very small mode effects for a more recent transition in Massachusetts. Second, we investigate the predictive evidence of validity for paper and online tests for predictions of future test scores and grades. We generally find minimal differences for the extent to which scores on paper tests can differentially predict future online versus paper test scores. Finally, online and paper test scores are similarly predictive of future grade point average. We conclude that the online test penalty can vary substantially by test and that extreme care should be taken when administering online tests to some students and paper tests to others.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

¹ Previous work spanning several states, including Massachusetts, suggests that test scores across different standards or assessment regimes provide similar information about students’ underlying ability (Backes et al., 2018).

² Boston, Worcester, and Springfield had the option of assigning individual schools to the online or paper format. Otherwise, districts selected a single test administration for the entire district. In November 2015, the Massachusetts State Board of Education voted to discontinue the PARCC assessment and implement a redeveloped version of the MCAS in all schools beginning in 2017.

³ Figures 1–3 originally displayed in Backes and Cowan (2019). These sample tests may be found at https://dc.mypearsonsupport.com/practice-tests/.

⁴ As noted below, because some specifications use twice-lagged test scores, the actual underlying data used are 2010 through 2018.

⁵ This conversion was performed by the Massachusetts Department of Elementary and Secondary Education in order to ensure that PARCC scores could be used for accountability purposes along with MCAS scores. Briefly, this process was conducted in two steps. First, comparable representative samples of MCAS and PARCC students were selected by ensuring that the prior performance and demographic variables for each sample looked similar to the overall mean from the final year that each student took the MCAS; i.e., 2014. Second, an equipercentile method was used to identify comparable test scores across the two different tests using student achievement percentiles for each test. For the full technical description, please see Massachusetts Department of Elementary and Secondary Education (2016).

⁶ For instance, test results under PARCC (2015 and 2016) are compared to test results using the MCAS 1.0 (2013 and 2014); similarly, test results under MCAS 2.0 (2017 and 2018) are compared to results using the PARCC (2015 and 2016).

⁷ All estimates were generated using StataCorp (2021).

⁸ Schools could obtain a waiver to administer the MCAS tests on paper, but waivers were granted only for cases in which online testing was not possible at a school (e.g., due to lack of technology required) or if every student in the school would take a paper-based test due to testing accommodations. For more information, see http://www.doe.mass.edu/news/news.aspx?id=25169.

Additional information

Funding

This research was funded by IES Research Grant R305A170119 and supported by the National Center for the Analysis of Longitudinal Data in Education Research (CALDER), which is funded by a consortium of foundations. For more information about CALDER funders, see www.caldercenter.org/about-calder. The data were provided by the state of Massachusetts and the state had the right to review the paper prior to public release in order to ensure that the paper does not disclose any personally identifiable information provided by the state. We thank Elana McDermott, Bob Lee, Dan Goldhaber, and three anonymous referees for their feedback. All errors are our own.