243
Views
0
CrossRef citations to date
0
Altmetric
Research Articles

iSTART: Adaptive Comprehension Strategy Training and Stealth Literacy Assessment

, , , , , , , & show all
Pages 2239-2252 | Received 03 Sep 2021, Accepted 10 Aug 2022, Published online: 02 Oct 2022

References

  • Afflerbach, P., Hurt, M., & Cho, B. Y. (2020). Reading comprehension strategy instruction. In D. L. Dinsmore, L. K. Fryer, & M. M. Parkinson (Eds.), Handbook of strategies and strategic processing (pp.98–118). Routledge.
  • Allen, L. K., & McNamara, D. S. (2015). You are your words: Modeling students’ vocabulary knowledge with natural language processing. In O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, & M. Desmarais (Eds.), Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015) (pp.258–265). International Educational Data Mining Society.
  • Allen, L. K., Snow, E. L., & McNamara, D. S. (2015). Are you reading my mind? Modeling students' reading comprehension skills with Natural Language Processing techniques. In J. Baron, G. Lynch, N. Maziarz, P. Blikstein, A. Merceron, & G. Siemens (Eds.), Proceedings of the 5th International Learning Analytics & Knowledge Conference (LAK'15) (pp. 246–254). ACM.
  • American College Testing. (2006). ACT high school profile report: The graduating class of 2006. https://www.act.org/content/dam/act/unsecured/documents/Natl-Scores-2006-National2006.pdf
  • Arner, T., McCarthy, K. S., & McNamara, D. S. (2021). iSTART StairStepper—Using comprehension strategy training to game the test. Computers, 10(4), 48. https://doi.org/10.3390/computers10040048
  • Baer, J. D., Cook, A. L., & Baldi, S. (2006). The literacy of America’s college students. American Institutes for Research.
  • Bai, C., Yang, J., & Tang, Y. (2022). Embedding self-explanation prompts to support learning via instructional video. Instructional Science, 1–21. https://doi.org/10.1007/s11251-022-09587-4
  • Baker, L., & Brown, A. L. (1984). Metacognitive skills and reading. In P. D. Pearson, R. Barr, M. Kamil, & P. Mosenthal (Eds.), Handbook of reading research (pp. 353–394). Longman.
  • Best, R. M., Rowe, M., Ozura, Y., & McNamara, D. S. (2005). Deep-level comprehension of science texts: The role of the reader and the text. Topics in Language Disorders, 25, 65–83.
  • Bowyer-Crane, C., & Snowling, M. J. (2005). Assessing children's inference generation: What do tests of reading comprehension measure? The British Journal of Educational Psychology, 75(Pt 2), 189–201.
  • Brown, A. L., Campione, J. C., & Day, J. D. (1981). Learning to learn: On training students to learn from texts. Educational Researcher, 10(2), 14–21. https://doi.org/10.3102/0013189X010002014
  • Butterfuss, R., Orcutt, E., Fang, Y., Kendeou, P., & McNamara, D. S. (2021, April 22–25). You pick’em: Selecting main ideas versus deleting details [Conference presentation]. American Educational Research Association (AERA) 2021 Annual Meeting.
  • Cain, K., & Oakhill, J. V. (1999). Inference making ability and its relation to comprehension failure in young children. Reading and Writing, 11(5/6), 489–503. https://doi.org/10.1023/A:1008084120205
  • Cain, K., Oakhill, J. V., & Elbro, C. (2003). The ability to learn new word meanings from context by school-age children with and without language comprehension difficulties. Journal of Child Language, 30(3), 681–694. https://doi.org/10.1017/S0305000903005713
  • Chen, X. (2013). STEM attrition: College students' paths into and out of STEM fields. Statistical analysis report [NCES 2014-001]. National Center for Education Statistics.
  • Chi, M. T., De Leeuw, N., Chiu, M. H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1207/s15516709cog1803_3
  • Coté, N., Goldman, S. R., & Saul, E. U. (1998). Students making sense of informational text: Relations between processing and representation. Discourse Processes, 25(1), 1–53. https://doi.org/10.1080/01638539809545019
  • Crossley, S., Kyle, K., Davenport, J., & McNamara, D. S. (2016). Automatic assessment of constructed response data in a chemistry tutor. International Educational Data Mining Society.
  • Davey, B., & McBride, S. (1986). Effects of question-generation training on reading comprehension. Journal of Educational Psychology, 78(4), 256–262. https://doi.org/10.1037/0022-0663.78.4.256
  • Elbro, C., & Buch-Iversen, I. (2013). Activation of background knowledge for inference making: Effects on reading comprehension. Scientific Studies of Reading, 17(6), 435–452. https://doi.org/10.1080/10888438.2013.774005
  • Ericsson, K. A. (2008). Deliberate practice and acquisition of expert performance: A general overview. Academic Emergency Medicine, 15(11), 988–994.
  • Fang, Y., Roscoe, R. D., & McNamara, D. S. (2021). Predicting reading skills via stealth assessment using educational games. Manuscript in preparation.
  • Fayer, S., Lacey, A., & Watson, A. (2017). STEM occupations: Past, present, and future. Spotlight on Statistics. https://stats.bls.gov/spotlight/2017/science-technology-engineering-and-mathematics-stem-occupations-past-present-and-future/pdf/science-technology-engineering-and-mathematics-stem-occupations-past-present-and-future.pdf.
  • Goldman, S. R., Braasch, J. L., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47(4), 356–381. https://doi.org/10.1002/RRQ.027
  • Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101(3), 371–395.
  • Gulikers, J. T. M., Kester, L., Kirschner, P. A., & Bastiaens, T. J. (2008). The effect of practice experience on perceptions of assessment authenticity, study approach, and learning outcomes. Learning and Instruction, 18(2), 172–186. https://doi.org/10.1016/j.learninstruc.2007.02.012
  • Healy, A. F., Schneider, V. I., & Bourne, L. E. Jr. (2012). Empirically valid principles of training. In A. F. Healy & L. E. Bourne, (Eds.), Training cognition: Optimizing efficiency, durability, and generalizability (pp. 13–39). Psychology Press.
  • Jackson, G. T., & McNamara, D. S. (2013). Motivation and performance in a game-based intelligent tutoring system. Journal of Educational Psychology, 105(4), 1036–1049. https://doi.org/10.1037/a0032580
  • Jackson, G. T., & McNamara, D. S. (2017). The motivation and mastery cycle framework: Predicting long-term benefits of educational games. In Y. Baek (Ed.), Game-based learning: Theory, strategies and performance outcomes (pp. 97–122). Nova Science Publishers.
  • Jackson, G. T., Dempsey, K. B., & McNamara, D. S. (2012). Game-based practice in a reading strategy tutoring system: Showdown in iSTART-ME. In H. Reinders (Ed.), Computer games (pp. 115–138). Multilingual Matters.
  • Jackson, G. T., Snow, E. L., Varner (Allen, L. K., & McNamara, D. S. (2013). Game performance as a measure of comprehension and skill transfer. In C. Boonthum-Denecke & G. M. Youngblood (Eds.), Proceedings of the 26th Annual Flordia Artificial Intelligence Research Society (FLAIRS) Conference (pp. 497–502). The AAAI Press.
  • Jackson, G. T., Varner, L. K., Denecke, C. B., & McNamara, D. S. (2013). The impact of individual differences on learning with an educational game and a traditional ITS. International Journal of Learning Technology, 8(4), 315–336. https://doi.org/10.1504/IJLT.2013.059129
  • Jackson, G. T., Boonthum, C., & McNamara, D. S. (2015). Natural language processing and game-based practice in iSTART. Journal of Interactive Learning Research, 26(2), 189–208.
  • Jackson, G. T., Boonthum, C., & McNamara, D. S. (2010). The efficacy of iSTART extended practice: Low ability students catch up. In J. Kay & V. Aleven (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (pp. 349–351). Springer.
  • Jacovina, M. E., Jackson, G. T., Snow, E. L., & McNamara, D. S. (2016). Timing game-based practice in a reading comprehension strategy tutor. In A. Micarelli, J. Stamper, & K. Panourgia (Eds.), Proceedings of the 13th International Conference on Intelligent Tutoring Systems (ITS 2016) (pp. 80–89). Springer.
  • James, L. T., & Casidy, R. (2018). Authentic assessment in business education: Its effects on student satisfaction and promoting behavior. Studies in Higher Education, 43(3), 401–415. https://doi.org/10.1080/03075079.2016.1165659
  • Johnson, A. M., Guerrero, T. A., Tighe, E. L., McNamara, D. S. (2017, June). iSTART-ALL: Confronting adult low literacy with intelligent tutoring for reading comprehension [Paper presentation]. International Conference on Artificial Intelligence in Education (pp. 125–136). Springer.
  • Johnson, A. M., Jacovina, M. E., Russell, D. E., & Soto, C. M. (2016). Challenges and solutions when using technologies in the classroom. In S. A. Crossley & D. S. McNamara (Eds.), Adaptive educational technologies for literacy instruction (pp. 13–29). Taylor & Francis.
  • Johnson, A. M., Ozogul, G., & Reisslein, M. (2015). Supporting multimedia learning with visual signalling and animated pedagogical agent: Moderating effects of prior knowledge. Journal of Computer Assisted Learning, 31(2), 97–115. https://doi.org/10.1111/jcal.12078
  • Johnson, A. M., McCarthy, K. S., Kopp, K., Perret, C. A., & McNamara, D. S. (2017). Adaptive reading and writing instruction in iSTART and W-Pal. In Z. Markov & V. Rus (Eds.), Proceedings of the 30th Annual Florida Artificial Intelligence Research Society International Conference (FLAIRS) (pp. 561–566). AAAI Press.
  • Kastberg, D., Chan, J. Y., & Murray, G. (2016). Performance of US 15-year-old students in science, reading, and mathematics literacy in an international context: First look at PISA 2015 [NCES 2017-048]. National Center for Education Statistics.
  • Kendeou, P. (2015). A general inference skill. In E. J. O’Brien, A. E. Cook, & R. F. Lorch (Eds.), Inferences during reading (pp. 160–181). Cambridge University Press.
  • Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95(2), 163–182. https://doi.org/10.1037/0033-295x.95.2.163
  • Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge University Press.
  • Long, D. L., Oppy, B. J., & Seely, M. R. (1994). Individual differences in the time course of inferential processing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20(6), 1456–1470. https://doi.org/10.1037/0278-7393.20.6.1456
  • MacGinitie, W. H., & MacGinitie, R. K. (1989). Gates–MacGinitie reading tests. Riverside.
  • Mayer, R. E. (2008). Applying the science of learning: Evidence-based principles for the design of multimedia instruction. The American Psychologist, 63(8), 760–769. https://doi.org/10.1037/0003-066X.63.8.760
  • Mayer, R. E. (2021). Evidence-based principles for how to design effective instructional videos. Journal of Applied Research in Memory and Cognition, 10(2), 229–240. https://doi.org/10.1016/j.jarmac.2021.03.007
  • McCarthy, K. S., Allen, L. K., Hinze, S. R. (2020). Predicting reading comprehension from constructed responses: Explanatory retrievals as stealth assessment [Paper presentation]. In Proceedings of the International Conference on Artificial Intelligence in Education (pp. 197–202). Springer.
  • McCarthy, K. S., Watanabe, M., & McNamara, D. S. (2020). The design implementation framework: Guiding principles for the redesign of a reading comprehension intelligent tutoring system. In M. Schmidt, A. Tawfik, Y. Earnshaw, & I. Jahnke (Eds.), Learner and User Experience Research: An introduction for the Field of Learning Design & Technology. EdTech Books. https://edtechbooks.org/ux/9_the_design_impleme
  • McCarthy, K. S., Watanabe, M., Dai, J., & McNamara, D. S. (2020). Personalized learning in iSTART: Past modifications and future design. Journal of Research on Technology in Education, 52(3), 301–321. https://doi.org/10.1080/15391523.2020.1716201
  • McNamara, D. S. (2004). SERT: Self-explanation reading training. Discourse Processes, 38(1), 1–30. https://doi.org/10.1207/s15326950dp3801_1
  • McNamara, D. S. (2009). The importance of teaching reading strategies. Perspectives on Language and Literacy, 35(2), 34–40.
  • McNamara, D. S. (2017). Self-explanation and reading strategy training (SERT) improves low-knowledge students’ science course performance. Discourse Processes, 54(7), 479–492. https://doi.org/10.1080/0163853X.2015.1101328
  • McNamara, D. S., & Healy, A. F. (1995). A procedural explanation of the generation effect: The use of an operand retrieval strategy for multiplication and addition problems. Journal of Memory and Language, 34(3), 399–416. https://doi.org/10.1006/jmla.1995.1018
  • McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–288. https://doi.org/10.1080/01638539609544975
  • McNamara, D. S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. In B. H. Ross (Ed.), The psychology of learning and motivation (Vol. 51, pp. 297–384). Academic Press.
  • McNamara, D. S., & McDaniel, M. A. (2004). Suppressing irrelevant information: Knowledge activation or inhibition? Journal of Experimental Psychology. Learning, Memory, and Cognition, 30(2), 465–482. https://doi.org/10.1037/0278-7393.30.2.465
  • McNamara, D. S., Jacovina, M. E., Snow, E. L., & Allen, L. K. (2015). From generating in the lab to tutoring systems in classrooms. The American Journal of Psychology, 128(2), 159–172. https://doi.org/10.5406/amerjpsyc.128.2.0159
  • McNamara, D. S., O’Reilly, T., Rowe, M., Boonthum, C., & Levinstein, I. B. (2007). iSTART: A web-based tutor that teaches self-explanation and metacognitive reading strategies. In D. S. McNamara (ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 397–420). Erlbaum.
  • McNamara, D. S., O'Reilly, T. P., Best, R. M., & Ozuru, Y. (2006). Improving adolescent students' reading comprehension with iSTART. Journal of Educational Computing Research, 34(2), 147–171. https://doi.org/10.2190/1RU5-HDTJ-A5C8-JVWE
  • McNamara, D. S., Ozuru, Y., & Floyd, R. G. (2011). Comprehension challenges in the fourth grade: The roles of text cohesion, text genre, and readers' prior knowledge. International Electronic Journal of Elementary Education, 4, 229–257.
  • McNamara, D. S., Graesser, A. C., & Louwerse, M. M. (2012). Sources of text difficulty: Across genres and grades. In J. P. Sabatini, E. Albro, & T. O’Reilly (Eds.), Measuring up: Advances in how we assess reading ability (pp. 89–116). R&L Education.
  • McCarthy, K. S., Likens, A. D., Kopp, K. J., Watanabe, M., Perret, C. A., & McNamara, D. S. (2018). The “LO”-down on grit: Non-cognitive trait assessments fail to predict learning gains in iSTART and W-Pal [Paper presentation]. Companion Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK’18), Sydney, Australia.
  • Meijer, R. R., & Nering, M. L. (1999). Computerized adaptive testing: Overview and introduction. Applied Psychological Measurement, 23(3), 187–194. https://doi.org/10.1177/01466219922031310
  • Oakhill, J. (1984). Inferential and memory skills in children's comprehension of stories. British Journal of Educational Psychology, 54(1), 31–39. https://doi.org/10.1111/j.2044-8279.1984.tb00842.x
  • Oakhill, J., Cain, K., & Elbro, C. (2019). Reading comprehension and reading comprehension difficulties. In D. Kilpatrick, R. Joshi, & R. Wagner (Eds.), Reading development and difficulties (pp. 83–115). Springer.
  • Perret, C. A., Johnson, A. M., McCarthy, K. S., Guerrero, T. A., & McNamara, D. S. (2017). StairStepper: An adaptive remedial iSTART module. In B. Boulay, R. Baker & E. Andre (Eds.), Proceedings of the 18th International Conference on Artificial Intelligence in Education (AIED)., (pp. 557–560). Springer.
  • Rosenshine, B., Meister, C., & Chapman, S. (1996). Teaching students to generate questions: A review of the intervention studies. Review of Educational Research, 66(2), 181–221. https://doi.org/10.3102/00346543066002181
  • Rowe, M., & McNamara, D. S. (2008). Inhibition needs no negativity: Negativity links in the construction-integration model. In V. Sloutsky, B. Love, & K. McRae (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 1777–1782). Cognitive Science Society.
  • Rupley, W. H., Blair, T. R., & Nichols, W. D. (2009). Effective reading instruction for struggling readers: The role of direct/explicit teaching. Reading & Writing Quarterly, 25(2–3), 125–138. https://doi.org/10.1080/10573560802683523
  • Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences. Cambridge University Press.
  • Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International Journal of Human-Computer Studies, 74, 14–31. https://doi.org/10.1016/j.ijhcs.2014.09.006
  • Shute, V. J., & Rahimi, S. (2020). Stealth assessment of creativity in a physics video game. Computers in Human Behavior, 116, 1–13. https://doi.org/10.1016/j.chb.2020.106647
  • Shute, V. J., & Ventura, M. (2013). Measuring and supporting learning in games: Stealth assessment. The MIT Press.
  • Shute, V. J., Ventura, M., & Kim, Y. J. (2013). Assessment and learning of qualitative physics in Newton's playground. The Journal of Educational Research, 106(6), 423–430. https://doi.org/10.1080/00220671.2013.832970
  • Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106–117. https://doi.org/10.1016/j.chb.2016.05.047
  • Sithole, A., Chiyaka, E. T., McCarthy, P., Mupinga, D. M., Bucklein, B. K., & Kibirige, J. (2017). Student attraction, Persistence and retention in STEM programs: Successes and continuing challenges. Higher Education Studies, 7(1), 46–59. https://doi.org/10.5539/hes.v7n1p46
  • Snow, C. (2002). Reading for understanding: Toward an R&D program in reading comprehension. RAND Education.
  • Snow, E. L., Jackson, G. T., Varner, L. K., & McNamara, D. S. (2013a). Investigating the effects of off-task personalization on system performance and attitudes within a game-based environment. In S. K. D'Mello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (pp. 272–275). Springer.
  • Snow, E. L., Jackson, G. T., Varner, L. K., & McNamara, D. S. (2013b). The impact of system interactions on motivation and performance. In Proceedings of the 15th International Conference on Human-Computer Interaction (HCII) (pp. 103–107). Springer.
  • Snow, E. L., Jacovina, M. E., Allen, L. K., Dai, J., & McNamara, D. S. (2014). Entropy: A stealth assessment of agency in learning environments. In J. Stamper, Z. Pardos, M. Mavrikis, & B. M. McLaren (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (pp. 241–244). International Educational Data Mining Society.
  • Sotiriadou, P., Logan, D., Daly, A., & Guest, R. (2020). The role of authentic assessment to preserve academic integrity and promote skill development and employability. Studies in Higher Education, 45(11), 2132–2148. https://doi.org/10.1080/03075079.2019.1582015
  • Stevens, E. A., Park, S., & Vaughn, S. (2019). A review of summarizing and main idea interventions for struggling readers in grades 3 through 12: 1978–2016. Remedial and Special Education, 40(3), 131–149. https://doi.org/10.1177/0741932517749940
  • Tsai, C. Y., Lin, H. S., & Liu, S. C. (2020). The effect of pedagogical GAME model on students' PISA scientific competencies. Journal of Computer Assisted Learning, 36(3), 359–369. https://doi.org/10.1111/jcal.12406
  • Van Den Broek, P., Rapp, D. N., & Kendeou, P. (2005). Integrating memory-based and constructionist processes in accounts of reading comprehension. Discourse Processes, 39(2–3), 299–316. https://doi.org/10.1080/0163853X.2005.9651685
  • VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.
  • Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29(6), 2568–2572. https://doi.org/10.1016/j.chb.2013.06.033
  • von der Embse, N., Jester, D., Roy, D., & Post, J. (2018). Test anxiety effects, predictors, and correlates: A 30-year meta-analytic review. Journal of Affective Disorders, 227, 483–493.
  • Wang, F., Li, W., Mayer, R. E., & Liu, H. (2018). Animated pedagogical agents as aids in multimedia learning: Effects on eye-fixations during learning and learning outcomes. Journal of Educational Psychology, 110(2), 250–268. https://doi.org/10.1037/edu0000221
  • Wolfe, M. B., & Goldman, S. R. (2005). Relations between adolescents' text processing and reasoning. Cognition and Instruction, 23(4), 467–502. https://doi.org/10.1207/s1532690xci2304_2
  • Xue, Y., Larson, R. C. (2015). Stem crisis or stem surplus? yes and yes: Monthly labor review. U.S. Bureau of Labor Statistics. Retrieved February 9, 2022, from https://www.bls.gov/opub/mlr/2015/article/stem-crisis-or-stem-surplus-yes-and-yes.htm

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.