Assessing Graph Comprehension on Paper and Computer with MBA Students: A Crossover Experimental Study

References

• ACT. (2013). The ACT Sample Science Questions: Passage 4. http://www.actstudent.org/sampletest/science/sci_04.html
   Google Scholar
• Alexander, M. W., Bartlett, J. E., Truell, A. D., & Ouwenga, K. (2001). Testing in a computer technology course: An investigation of equivalency in performance between online and paper and pencil methods. Journal of Career and Technical Education, 18(1), 69–21. https://doi.org/https://doi.org/10.21061/jcte.v18i1.600
   Google Scholar
• Anakwe, B. (2008). Comparison of student performance in paper-based versus computer-based testing. Journal of Education for Business, 84(1), 13–17. https://doi.org/https://doi.org/10.3200/JOEB.84.1.13-17
   Google Scholar
• Bennett, R. E. (2015). The changing nature of educational assessment. Review of Research in Education, 39(1), 370–407. https://doi.org/https://doi.org/10.3102/0091732X14554179
   Web of Science ®Google Scholar
• Bennett, R. E., Braswell, J., Oranje, A., Sandene, B., Kaplan, B., & Yan, F. (2008). Does it matter if I take my mathematics test on computer? A second empirical study of mode effects in NAEP. The Journal of Technology, Learning, and Assessment, 6(9), 1–37.
   Google Scholar
• Berg, C. A., & Boote, S. K. (2017). Format effects of empirically derived multiple-choice versus free-response instruments when assessing graphing abilities. International Journal of Science and Mathematics Education, 15(1), 19–38. https://doi.org/https://doi.org/10.1007/s10763-015-9678-6
   Web of Science ®Google Scholar
• Berg, C. A., & Smith, P. (1994). Assessing students’ abilities to construct and interpret line graphs: Disparities between multiple-choice and free-response instruments. Science Education, 78(6), 527–554. https://doi.org/https://doi.org/10.1002/sce.3730780602
   Web of Science ®Google Scholar
• Bertin, J. 1973. Semiologie Graphique. W. Berg & H. Wainer, 2nd. The Hague: Mouton–Gautier.
   Google Scholar
• Bertin, J. (1983). Semiology of graphics: Diagrams, networks, maps (W. J. Berg, Trans.). The University of Wisconsin Press, Ltd.
   Google Scholar
• Board, C. (2015). Test specifications for the redesigned SAT®.
   Google Scholar
• Boote, S. K. (2014). Assessing and understanding line graph interpretations using a scoring rubric of organized cited factors. Journal of Science Teacher Education, 25(3), 333–354. https://doi.org/https://doi.org/10.1007/s10972-012-9318-8
   Google Scholar
• Boote, S. K., & Boote, D. N. (2017). Leaping from discrete to continuous independent variables: Sixth graders’ science line graph interpretations. The Elementary School Journal, 117(3), 455–484. https://doi.org/https://doi.org/10.1086/690204
   Web of Science ®Google Scholar
• Broadfoot, P., & Black, P. (2004). Redefining assessment? The first ten years of Assessment in Education. Assessment in Education: Principles, Policy & Practice, 11(1), 7–26.
   Google Scholar
• Brunet Marks, A., & Moss, S. A. (2016). What predicts law student success? A longitudinal study correlating law student applicant data and law school outcomes. Journal of Empirical Legal Studies, 13(2), 205–265. https://doi.org/https://doi.org/10.1111/jels.12114
   Web of Science ®Google Scholar
• Buerger, S., Kroehne, U., & Goldhammer, F. (2016). The transition to computer-based testing in large-scale assessments: Investigating (partial) measurement invariance between modes. Psychological Test and Assessment Modeling, 58(4), 597–616.
   Google Scholar
• Card, S. K. (2009). Information visualization. In A. Sears & J. A. Jacko (Eds.), Human-computer interaction: Design issues, solutions, and applications (pp. 182–216). Taylor & Francis Group.
   Google Scholar
• Carmody, T. (2012). Why education publishing is big business. WIRED. https://www.wired.com/2012/01/why-education-publishing-is-big-business
   Google Scholar
• Carpenter, P. A., & Shah, P. (1998). A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology. Applied, 4(2), 75–100. https://doi.org/https://doi.org/10.1037/1076-898X.4.2.75
   Web of Science ®Google Scholar
• Chen, J., & Perie, M. (2018). Comparability within Computer-Based Assessment: Does Screen Size Matter? Computers in the Schools, 35(4), 268–283. https://doi.org/https://doi.org/10.1080/07380569.2018.1531599
   Web of Science ®Google Scholar
• Clariana, R. B., & Wallace, P. (2002). Paper–based versus computer–based assessment: Key factors associated with the test mode effect. British Journal of Educational Technology, 33(5), 593–602. https://doi.org/https://doi.org/10.1111/1467-8535.00294
   Web of Science ®Google Scholar
• Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications, Inc.
   Google Scholar
• Debeer, D., & Janssen, R. (2013). Modeling item‐position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164–185. https://doi.org/https://doi.org/10.1111/jedm.12009
   Web of Science ®Google Scholar
• DeLeeuw, K. E., & Mayer, R. E. (2008). A comparison of three measures of cognitive load: Evidence for separable measures of intrinsic, extraneous, and germane load. Journal of Educational Psychology, 100(1), 223–234. https://doi.org/https://doi.org/10.1037/0022-0663.100.1.223
   Web of Science ®Google Scholar
• Dillon, A. (1992). Reading from paper versus screens: A critical review of the empirical literature. Ergonomics, 35(10), 1297–1326. https://doi.org/https://doi.org/10.1080/00140139208967394
   Web of Science ®Google Scholar
• Donnelly-Hermosillo, D. F., Gerard, L. F., & Linn, M. C. (2020). Impact of graph technologies in K-12 science and mathematics education. Computers & Education, 146, 1–32. https://doi.org/https://doi.org/10.1016/j.compedu.2019.103748
   Web of Science ®Google Scholar
• Draeger, R. (2009). Within-Subjects Design. In M. Allen (Ed.), The SAGE Encyclopedia of Communication Research Methods (pp. 1878–1880). Sage.
   Google Scholar
• Dreben, E. K., Fiske, S. T., & Hastie, R. (1979). The independence of evaluative and item information: Impression and recall order effects in behavior-based impression formation. Journal of Personality and Social Psychology, 37(10), 1758–1768. https://doi.org/https://doi.org/10.1037/0022-3514.37.10.1758
   Web of Science ®Google Scholar
• ETS-GRE. (2018). GRE® General test: Quantitative reasoning question types. https://www.ets.org/gre/revised_general/about/content/quantitative_reasoning
   Google Scholar
• Friel, S. N., Curcio, F. R., & Bright, G. W. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32(2), 124–158. https://doi.org/https://doi.org/10.2307/749671
   Web of Science ®Google Scholar
• Gallagher, A., Bridgeman, B., & Cahalan, C. (2000). The effect of computer-based tests on racial/ethnic, gender, and language groups (GRE Board Professional Report No. 96–21P). Education Testing Service.
   Google Scholar
• Garas, S., & Hassan, M. (2018). Student performance on computer-based tests versus paper-based tests in introductory financial accountig: UAE evidence. Academy of Accounting and Financial Studies Journal, 22(2), 1–14.
   Google Scholar
• Goldberg, A. L., & Pedulla, J. J. (2002). Performance differences according to test mode and computer familiarity on a practice Graduate Record Exam. Educational and Psychological Measurement, 62(6), 1053–1067. https://doi.org/https://doi.org/10.1177/0013164402238092
   Web of Science ®Google Scholar
• Graduate Management Admissions Council. (2012). GMAT Review (13th ed.). John Wiley & Sons, Inc.
   Google Scholar
• Greeno, J. G., & Hall, R. P. (1997). Practicing representation: Learning with and about representational forms. Phi Delta Kappan, 78(5), 361–367.
   Web of Science ®Google Scholar
• Groth, R. E. (2007). Toward a conceptualization of statistical knowledge for teaching. Journal for Research in Mathematics Education, 38(5), 427–437.
   Web of Science ®Google Scholar
• Hambleton, R. K., & Traub, R. E. (1974). The effects of item order on test performance and stress. The Journal of Experimental Education, 43(1), 40–46. https://doi.org/https://doi.org/10.1080/00220973.1974.10806302
   Web of Science ®Google Scholar
• Heer, J., & Shneiderman, B. (2012). Interactive dynamics: A taxonomy of tools that support the fluent and flexible use of visualizations. Queue, 10(2), 1–26. https://doi.org/https://doi.org/10.1145/2133416.2146416
   Google Scholar
• Hegarty, M. (2011). The cognitive science of Visual‐Spatial displays: Implications for design. Topics in Cognitive Science, 3(3), 446–474. https://doi.org/https://doi.org/10.1111/j.1756-8765.2011.01150.x
   PubMed Web of Science ®Google Scholar
• Herrington, J. D. (2010). MBA: Past, present and future. Academy of Educational Leadership Journal, 14(1), 63–76.
   Google Scholar
• Higgins, J., Russell, M., & Hoffmann, T. (2005). Examining the effect of computer-based passage presentation on reading test performance. The Journal of Technology, Learning, and Assessment, 3(4), 1–35.
   Google Scholar
• Hilbert, S., & Jaffe, E. D. (2012). Barron’s new GMAT: Graduate Management Admission Test (17th ed.). Barron’s Educational Series, Inc.
   Google Scholar
• Hohensinn, C., Kubinger, K. D., Reif, M., Holocher-Ertl, S., Khorramdel, L., & Frebort, M. (2008). Examining item-position effects in large-scale assessment using the linear logistic test model. Psychology Science Quarterly, 50(3), 391–402.
   Google Scholar
• Hutchins, E. (1995). Cognition in the wild. MIT Press.
   Google Scholar
• Hutchins, E. (2010). Cognitive ecology. Topics in Cognitive Science, 2(4), 705–715. https://doi.org/https://doi.org/10.1111/j.1756-8765.2010.01089.x
   PubMed Web of Science ®Google Scholar
• Jang, J., Trickett, S. B., Schunn, C. D., & Trafton, J. G. (2012). Unpacking the temporal advantage of distributing complex visual displays. International Journal of Human-Computer Studies, 70(11), 812–827. https://doi.org/https://doi.org/10.1016/j.ijhcs.2012.07.003
   Web of Science ®Google Scholar
• Kingston, N. M. (2009a). Comparability of computer- and paper-administered multiple-choice tests for K-12 populations: A synthesis. Applied Measurement in Education, 22(1), 22–37. https://doi.org/https://doi.org/10.1080/08957340802558326
   Web of Science ®Google Scholar
• Kingston, N. M., & Dorans, N. J. (1984). Item location effects and their implications for IRT equating and adaptive testing. Applied Psychological Measurement, 8(2), 147–154. https://doi.org/https://doi.org/10.1177/014662168400800202
   Web of Science ®Google Scholar
• Konold, C., Higgins, T., Russell, S. J., & Khalil, K. (2015). Data seen through different lenses. Educational Studies in Mathematics, 88(3), 305–325. https://doi.org/https://doi.org/10.1007/s10649-013-9529-8
   Web of Science ®Google Scholar
• Kuncel, N. R., Credé, M., & Thomas, L. L. (2007). A meta-analysis of the predictive validity of the Graduate Management Admission Test (GMAT) and undergraduate grade point average (UGPA) for graduate student academic performance. Academy of Management Learning & Education, 6(1), 51–68. https://doi.org/https://doi.org/10.5465/amle.2007.24401702
   Web of Science ®Google Scholar
• Kuo, C.-Y., & Wu, H.-K. (2013). Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science. Computers & Education, 68, 388–403. https://doi.org/https://doi.org/10.1016/j.compedu.2013.06.002
   Web of Science ®Google Scholar
• Lai, K., Cabrera, J., Vitale, J. M., Madhok, J., Tinker, R., & Linn, M. C. (2016). Measuring graph comprehension, critique, and construction in science. Journal of Science Education and Technology, 25(4), 665–681. https://doi.org/https://doi.org/10.1007/s10956-016-9621-9
   Web of Science ®Google Scholar
• Latour, B. (1986). Visualization and cognition: Thinking with eyes and hands. Knowledge and Society, 6(6), 1–40. http://hci.ucsd.edu/10/readings/Latour(1986).pdf
   Google Scholar
• Lead States, N. G. S. S. (2013). Next generation science standards: For states, by states. The National Academies Press.
   Google Scholar
• Lee, J., & Hopkins, L. (1985). The effects of training on computerized aptitude test performance and anxiety. Paper presented at the The Annual Meeting of the Eastern Psychological Association, Boston, MA.
   Google Scholar
• Lee, J., Moreno, K. E., & Sympson, J. B. (1986). The effects of mode of test administration on test performance. Educational and Psychological Measurement, 46(2), 467–474. https://doi.org/https://doi.org/10.1177/001316448604600224
   Web of Science ®Google Scholar
• Leeson, H. V. (2006). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Testing, 6(1), 1–24. https://doi.org/https://doi.org/10.1207/s15327574ijt0601_1
   Google Scholar
• Lissitz, R. W., & Jiao, H. (Eds.). (2012). Computers and their impact on state assessments: Recent history and predictions for the future. Information Age Publishing, Inc.
   Google Scholar
• MacCann, R. S. (2006). The equivalence of online and traditional testing for different subpopulations and item types. British Journal of Educational Technology, 37(1), 79–91. https://doi.org/https://doi.org/10.1111/j.1467-8535.2005.00524.x
   Web of Science ®Google Scholar
• Mangen, A., Walgermo, B. R., & Brønnick, K. (2013). Reading linear texts on paper versus computer screen: Effects on reading comprehension. International Journal of Educational Research, 58, 61–68. https://doi.org/https://doi.org/10.1016/j.ijer.2012.12.002
   Web of Science ®Google Scholar
• Mazzeo, J., & Harvey, A. L. (1988). The equivalence of scores from conventional and automated educational and psychological tests: A review of literature. Educational Testing Service.
   Google Scholar
• Meletiou-Mavrotheris, M., & Lee, C. (2010). Investigating college-level introductory statistics students’ prior knowledge of graphing. Canadian Journal of Science, Mathematics and Technology Education, 10(4), 339–355. https://doi.org/https://doi.org/10.1080/14926156.2010.524964
   Google Scholar
• Mitchell, K., Lewis, R. S., Satterfield, J., & Hong, B. A. (2016). The new Medical College Admission Test: Implications for teaching psychology. American Psychologist, 71(2), 125–135. https://doi.org/https://doi.org/10.1037/a0039975
   PubMed Web of Science ®Google Scholar
• National Council for Social Studies. (2010). National curriculum standards for social studies: A framework for teaching, learning, and assessment Retrieved from
   Google Scholar
• National Governors Association Center for Best Practice & Council of Chief State School Officers. (2010). Common Core State Standards.
   Google Scholar
• National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press.
   Google Scholar
• Noyes, J. M., Garland, K. J., & Robbins, E. (2004). Paper-based versus computer-based assessment—Is workload another test mode effect? British Journal of Educational Technology, 35(1), 111–113. https://doi.org/https://doi.org/10.1111/j.1467-8535.2004.00373.x
   Web of Science ®Google Scholar
• Oh, I.-S., Schmidt, F. L., Shaffer, F. A., & Le, H. (2008). The Graduate Management Admission Test (GMAT) is even more valid than we thought: A new development in meta-analysis and its implications for the validity of the GMAT. Academy of Management Learning & Education, 7(4), 563–570. https://doi.org/https://doi.org/10.5465/amle.2008.35882196
   Web of Science ®Google Scholar
• Paas, F., Van Gog, T., & Sweller, J. (2010). Cognitive load theory: New conceptualizations, specifications, and integrated research perspectives. Educational Psychology Review, 22(2), 123–138. https://doi.org/https://doi.org/10.1007/s10648-010-9133-8
   Web of Science ®Google Scholar
• Padilla, L. M., Creem-Regehr, S. H., Hegarty, M., & Stefanucci, J. K. (2018). Decision making with visualizations: A cognitive framework across disciplines. Cognitive Research: Principles and Implications, 3(29), 1–25. https://doi.org/https://doi.org/10.1186/s41235-018-0120-9
   PubMed Web of Science ®Google Scholar
• Parshall, C. G., & Kromrey, J. D. (1993). Computer testing versus paper-and-pencil testing: An analysis of examinee characteristics associated with mode effect. Paper presented at the Annual Meeting of the American Educational Research Association Atlanta, GA.
   Google Scholar
• Phage, I. B., Lemmer, M., & Hitge, M. (2017). Probing factors influencing students’ graph comprehension regarding four operations in kinematics graphs. African Journal of Research in Mathematics, Science and Technology Education, 21(2), 200–210. https://doi.org/https://doi.org/10.1080/18117295.2017.1333751
   Web of Science ®Google Scholar
• Poggio, J., Glasnapp, D. R., Yang, X., & Poggio, A. J. (2005). A comparative evaluation of score results from computerized and paper & pencil mathematics testing in a large scale state assessment program. Journal of Technology, Learning, and Assessment, 3(6), 1–29.
   Google Scholar
• Rodriguez, M. C. (2013). Trends in graduate admissions measures. Paper presented at the BUROS Big Issues in Testing Conference, Lincoln, NE.
   Google Scholar
• Russell, M., Goldberg, A., & O’Connor, K. (2003). Computer-based testing and validity: A look back into the future. Assessment in Education: Principles, Policy & Practice, 10(3), 279–293.
   Google Scholar
• Sandene, B., Horkay, N., Bennett, R., Allen, N., Kaplan, B., & Oranje, A. (2005). Online assessment in mathematics and writing: Reports from NAEP technology-based assessment project. Retrieved from Washington, DC:
   Google Scholar
• Schoenfeld, A. H. (2008). Problem solving in the United States, 1970-2008: Research and theory, practice and politics. ZDM Mathematics Education, 39(5–6), 537–551. https://doi.org/https://doi.org/10.1007/s11858-007-0038-z
   Google Scholar
• Schroeders, U., & Wilhelm, O. (2011). Equivalence of reading and listening comprehension across test media. Educational and Psychological Measurement, 71(5), 849–869. https://doi.org/https://doi.org/10.1177/0013164410391468
   Web of Science ®Google Scholar
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
   Google Scholar
• Shah, P., & Hoeffner, J. (2002). Review of graph comprehension research: Implications for instruction. Educational Psychology Review, 14(1), 47–69. https://doi.org/https://doi.org/10.1023/A:1013180410169
   Web of Science ®Google Scholar
• Shah, P., Meyer, R. E., & Hegarty, M. (1999). Graphs as aids to knowledge construction: Signaling techniques for guiding the process of graph comprehension. Journal of Educational Psychology, 91(4), 690–702. https://doi.org/https://doi.org/10.1037/0022-0663.91.4.690
   Web of Science ®Google Scholar
• Shaughnessy, J. M. (2007). Research on statistical learning and reasoning. In F. K. Lester (Ed.), Second Handbook of Research on Mathematics Teaching and Learning (Vol. 2, pp. 957–1010). Information Age Publishing.
   Google Scholar
• Singer, L. M., & Alexander, P. A. (2017). Reading on paper and digitally: What the past decades of empirical research reveal. Review of educational research, 87(6), 1007-1041.
   Google Scholar
• Strobel, B., Grund, S., & Lindner, M. A. (2018). Do seductive details do their damage in the context of graph comprehension? Insights from eye movements. Applied Cognitive Psychology, 33(1), 95–108. https://doi.org/https://doi.org/10.1002/acp.3491
   Web of Science ®Google Scholar
• Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/https://doi.org/10.1207/s15516709cog1202_4
   Web of Science ®Google Scholar
• Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12(3), 185–233. https://doi.org/https://doi.org/10.1207/s1532690xci1203_1
   Web of Science ®Google Scholar
• Tufte, E. R. (2001). The visual display of quantitative information (2nd Ed. ed.). Graphics Press.
   Google Scholar
• Wainer, H. (1992). Understanding graphs and tables. Educational Researcher, 21(1), 14–23. https://doi.org/https://doi.org/10.3102/0013189X021001014
   Google Scholar
• Wao, J. O., Ries, R., Flood, I., Lavy, S., & Ozbek, M. E. (2016). Relationship between admission GRE scores and graduation GPA scores of construction management graduate students. International Journal of Construction Education and Research, 12(1), 37–53. https://doi.org/https://doi.org/10.1080/15578771.2015.1050562
   Google Scholar
• Watson, B. (2001). Key factors affecting conceptual gains from CAL material. British Journal of Education Technology, 32(587-593).
   Google Scholar