Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning

REFERENCES

• Afzal, S., & Robinson, P. (2011). Natural affect data: Collection and annotation. In R. Calvo & S. D'Mello (Eds.), New perspectives on affect and learning technologies (pp. 44–70). New York, NY: Springer.
   Google Scholar
• Arroyo, I., Beal, C. R., Murray, T., Walles, R., & Woolf, B. P. (2004). Web-based intelligent multimedia tutoring for high stakes achievement tests. In J. C. Lester, R. M. Vicari, & F. Paraguaçu (Eds.), Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004) (pp. 468–477). Berlin, Germany: Springer.
   Google Scholar
• Arroyo, I., Woolf, B., Cooper, D., Burleson, W., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. In V. Dimitrova, R. Mizoguchi, B. Du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 17–24). Amsterdam, The Netherlands: IOS Press.
   Google Scholar
• Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94.
   Web of Science ®Google Scholar
• Baker, R., & Ocumpaugh, J. (2015). Interaction-based affect detection in educational software. In R. Calvo, S. D'Mello, J. Gratch, & A. Kappas (Eds.), The Oxford handbook of affective computing (pp. 233–245). New York, NY: Oxford University Press.
   Google Scholar
• Baker, R., & Rossi, L. M. (2013). Assessing the disengaged behaviors of learners. In R. Sottilare, A. Graesser, X. Hu, & H. Holden (Eds.), Design recommendations for intelligent tutoring systems (pp. 155–165). Orlando, FL: Army Research Laboratory.
   Google Scholar
• Baker, R. S., & Ocumpaugh, J. (2014). Cost-effective, actionable engagement detection at scale. In Proceedings of the 7th International Conference on Educational Data Mining (pp. 345–346). Retrieved from http://educationaldatamining.org/EDM2014/uploads/procs2014/posters/21_EDM-2014-Poster.pdf
   Google Scholar
• Baker, R. S. J., Kalka, J., Aleven, V., Rossi, L., Gowda, S. M., Wagner, A. Z., … Ocumpaugh, J. (2012). Towards sensor-free affect detection in cognitive tutor algebra. In K. Yacef, O. Zaïane, H. Hershkovitz, M. Yudelson, & J. Stamper (Eds.), Proceedings of the 5th International Conference on Educational Data Mining (pp. 126–133). Retrieved from http://www.educationaldatamining.org/conferences/index.php/EDM/2012/paper/view/967/933
   Google Scholar
• Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.
   Google Scholar
• Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: Freeman.
   Google Scholar
• Barrett, L. (2006). Are emotions natural kinds? Perspectives on Psychological Science, 1, 28–58.
   Web of Science ®Google Scholar
• Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59(1), 617–645.
   PubMed Web of Science ®Google Scholar
• Bartlett, M. S., Littlewort, G. C., Frank, M. G., & Lee, K. (2014). Automatic decoding of facial movements reveals deceptive pain expressions. Current Biology, 24, 738–743.
   PubMed Web of Science ®Google Scholar
• Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., & Craven, P. L. (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine, 78( Suppl. 1), B231–B244.
   PubMedGoogle Scholar
• Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205–212.
   Google Scholar
• Bixler, R., & D'Mello, S. (2013). Detecting engagement and boredom during writing with keystroke analysis, task appraisals, and stable traits. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (IUI 2013) (pp. 225–234). New York, NY: ACM.
   Google Scholar
• Bixler, R., & D'Mello, S. K. (2016). Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Modeling & User-Adapted Interaction, 26(1), 33–68.
   Web of Science ®Google Scholar
• Blanchard, N., Bixler, R., Joyce, T., & D'Mello, S. K. (2014). Automated physiological-based detection of mind wandering during learning. In S. Trausan-Matu, K. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th international conference on intelligent tutoring systems (pp. 55–60). Berlin: Springer.
   Google Scholar
• Bosch, N., Chen, H., Baker, R., Shute, V., & D'Mello, S. K. (2015). Accuracy vs. availability heuristic in multimodal affect detection in the wild. Proceedings of the 17th ACM international conference on multimodal interaction (ICMI 2015). New York, NY: ACM.
   Google Scholar
• Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., & Shute, V. (2016). Using video to automatically detect learner affect in computer-enabled classrooms. ACM Transactions on Interactive Intelligent Systems, 6(2), 17.1–17.31.
   PubMed Web of Science ®Google Scholar
• Bulger, M. E., Mayer, R. E., Almeroth, K. C., & Blau, S. D. (2008). Measuring learner engagement in computer-equipped college classrooms. Journal of Educational Multimedia and Hypermedia, 17, 129–143.
   Google Scholar
• Calvo, R., D'Mello, S. K., Gratch, J., & Kappas, A. (Eds.). (2015). The oxford handbook of affective computing. New York, NY: Oxford University Press.
   Google Scholar
• Camras, L., & Shutter, J. (2010). Emotional facial expressions in infancy. Emotion Review, 2, 120–129.
   Web of Science ®Google Scholar
• Camras, L. A., & Witherington, D. C. (2005). Dynamical systems approaches to emotional development. Developmental Review, 25, 328–350. http://dx.doi.org/10.1016/j.dr.2005.10.002
   Web of Science ®Google Scholar
• Chi, M., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219–243.
   Web of Science ®Google Scholar
• Christenson, S. L., Reschly, A. L., & Wylie, C. (2012). Handbook of research on student engagement. New York, NY: Springer.
   Google Scholar
• Coan, J. A. (2010). Emergent ghosts of the emotion machine. Emotion Review, 2, 274–285.
   Web of Science ®Google Scholar
• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
   Web of Science ®Google Scholar
• Conati, C. (2002). Probabilistic assessment of user's emotions in educational games. Applied Artificial Intelligence, 16, 555–575. http://dx.doi.org/10.1080/08839510290030390
   Web of Science ®Google Scholar
• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273–297.
   Web of Science ®Google Scholar
• Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the experience-sampling method. The Journal of Nervous and Mental Disease, 175, 526–536.
   PubMed Web of Science ®Google Scholar
• D'Mello, S., Craig, S., Witherspoon, A., McDaniel, B., & Graesser, A. (2008). Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1–2), 45–80.
   Web of Science ®Google Scholar
• D'Mello, S., Dale, R., & Graesser, A. (2012). Disequilibrium in the mind, disharmony in the body. Cognition & Emotion, 26, 362–374. http://dx.doi.org/10.1080/02699931.2011.613668
   PubMed Web of Science ®Google Scholar
• D'Mello, S., & Graesser, A. (2011). The half-life of cognitive-affective states during complex learning. Cognition & Emotion, 25, 1299–1308.
   PubMed Web of Science ®Google Scholar
• D'Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22, 145–157. http://dx.doi.org/10.1016/j.learninstruc.2011.10.001
   Web of Science ®Google Scholar
• D'Mello, S. K. (2016). On the influence of an iterative affect annotation approach on inter-observer and self-observer reliability. IEEE Transactions on Affective Computing, 7, 136–149.
   Web of Science ®Google Scholar
• D'Mello, S. K., Dowell, N., & Graesser, A. C. (2013). Unimodal and multimodal human perception of naturalistic non-basic affective states during Human-Computer interactions. IEEE Transactions on Affective Computing, 4, 452–465.
   Web of Science ®Google Scholar
• D'Mello, S. K., & Graesser, A. C. (2014). Confusion. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 289–310). New York, NY: Routledge.
   Google Scholar
• D'Mello, S. K., Kopp, K., Bixler, R., & Bosch, N. (2016). Attending to attention: Detecting and combating mind wandering during computerized reading. In Extended abstracts of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2016) (pp. 1661–1669). New York, NY: ACM.
   Google Scholar
• D'Mello, S. K., & Kory, J. (2015). A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys, 47(3), 43:41–43:46.
   Web of Science ®Google Scholar
• D'Mello, S. K., Taylor, R., Davidson, K., & Graesser, A. (2008). Self versus teacher judgments of learner emotions during a tutoring session with AutoTutor. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 9–18). Berlin, Heidelberg: Springer-Verlag.
   Google Scholar
• Darwin, C. (1872). The expression of the emotions in man and animals. London, UK: John Murray.
   Google Scholar
• Daschmann, E. C., Goetz, T., & Stupnisky, R. H. (2011). Testing the predictors of boredom at school: Development and validation of the precursors to boredom scales. British Journal of Educational Psychology, 81, 421–440.
   PubMed Web of Science ®Google Scholar
• De Falco, J., Baker, R., & D'Mello, S. (2014). Addressing behavioral disengagement in online learning. In R. Sottilare, A. Graesser, X. Hu, & B. Goldberg (Eds.), Design recommendations for adaptive intelligent tutoring systems: Adaptive instructional strategies (Vol. 2, pp. 49–56). Orlando, FL: Army Research Laboratory.
   Google Scholar
• Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. New York, NY: Plenum.
   Google Scholar
• Deubel, H., & Schneider, W. X. (1996). Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision research, 36, 1827–1837.
   PubMed Web of Science ®Google Scholar
• deVega, M., Glenberg, A., & Graesser, A. (Eds.). (2008). Symbols, embodiment, and meaning. Oxford: Oxford University Press.
   Google Scholar
• Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87.
   Web of Science ®Google Scholar
• Drummond, J., & Litman, D. (2010). In the zone: Towards Detecting student zoning out using supervised machine learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (Vol. 6095, pp. 306–308). Berlin, Germany: Springer-Verlag.
   Google Scholar
• Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44, 237–251.
   PubMed Web of Science ®Google Scholar
• Eastwood, J. D., Frischen, A., Fenske, M. J., & Smilek, D. (2012). The unengaged mind: Defining boredom in terms of attention. Perspectives on Psychological Science, 7, 482–495.
   Web of Science ®Google Scholar
• Eccles, J., & Wang, M. T. (2012). Part I commentary: So what is student engagement anyway? In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 133–145). New York, NY: Springer.
   Google Scholar
• Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132.
   PubMed Web of Science ®Google Scholar
• Ekman, P. (1984). Expression and the nature of emotion. In K. Scherer & P. Ekman (Eds.), Approaches to emotion (pp. 319–344). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6, 169–200.
   Web of Science ®Google Scholar
• Ekman, P. (2002, November). Darwin, deception, and facial expression. Paper presented at the conference on Emotions Inside Out, 130 Years after Darwin's The Expression of the Emotions in Man and Animals, New York, NY.
   Google Scholar
• Ekman, P., & Friesen, W. (1978). The Facial Action Coding System: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.
   Google Scholar
• Elfenbein, H., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, 203–235. http://dx.doi.org/10.1037//0033-2909.128.2.203
   PubMed Web of Science ®Google Scholar
• Eyben, F., Wöllmer, M., & Schuller, B. (2010). Opensmile: The Munich versatile and fast open-source audio feature extractor. Proceedings of the International Conference on Multimedia (pp. 1459–1462). New York, NY: ACM.
   Google Scholar
• Faber, M., Bixler, R., & D'Mello, S. K. (in press). An automated behavioral measure of mind wandering rates during computerized reading. Behavior Research Methods.
   Google Scholar
• FACET. (2014). Facial Expression Recognition Software. Boston, MA: Emotient. Retrieved from https://imotions.com/emotient/
   Google Scholar
• Fiedler, K., & Beier, S. (2014). Affect and cognitive processes in educational contexts. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 36–56). New York, NY: Routledge.
   Google Scholar
• Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109.
   Web of Science ®Google Scholar
• Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). New York, NY: Springer.
   Google Scholar
• Gobert, J. D., Baker, R. S., & Wixon, M. B. (2015). Operationalizing and detecting disengagement within online science microworlds. Educational Psychologist, 50(1), 43–57.
   Web of Science ®Google Scholar
• Graesser, A., Chipman, P., Haynes, B., & Olney, A. (2005). AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48, 612–618. http://dx.doi.org/10.1109/TE.2005.856149
   Web of Science ®Google Scholar
• Graesser, A., McDaniel, B., Chipman, P., Witherspoon, A., D'Mello, S. K., & Gholson, B. (2006). Detection of emotions during learning with AutoTutor. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 285–290). Austin, TX: Cognitive Science Society.
   Google Scholar
• Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist, 50(1), 1–17.
   Web of Science ®Google Scholar
• Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36.
   PubMed Web of Science ®Google Scholar
• Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.
   Google Scholar
• Heine, S. J., Lehman, D. R., Peng, K., & Greenholtz, J. (2002). What's wrong with cross-cultural comparisons of subjective Likert scales?: The reference-group effect. Journal of Personality and Social Psychology, 82, 903–918.
   PubMed Web of Science ®Google Scholar
• Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technology-mediated learning: A review. Computers & Education, 90, 36–53.
   Web of Science ®Google Scholar
• Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41, 111–127.
   Web of Science ®Google Scholar
• Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Attention, Perception, & Psychophysics, 57, 787–795.
   PubMed Web of Science ®Google Scholar
• Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J., D'Mello, S K. (2017). Out of the fr-“eye”-ing pan: Towards gaze-based models of attention with learning technology in the classroom. Manuscript submitted for publication.
   Google Scholar
• Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004). A survey method for characterizing daily life experience: The day reconstruction method. Science, 306(5702), 1776–1780.
   PubMed Web of Science ®Google Scholar
• Kapoor, A., & Picard, R. (2005). Multimodal affect recognition in learning environments. In Proceedings of the 13th annual ACM International Conference on Multimedia (pp. 677–682). New York, NY: ACM.
   Google Scholar
• Kappas, A. (2013). Social regulation of emotion: messy layers. Frontiers in Psychology, 4(51). http://dx.doi.org/10.3389/fpsyg.2013.00051
   PubMed Web of Science ®Google Scholar
• Keltner, D., & Ekman, P. (2000). Facial expression of emotion. In R. Lewis & J. M. Haviland-Jones (Eds.), Handbook of emotions (Vol. 2nd ed, pp. 236–264). New York, NY: Guilford.
   Google Scholar
• Kory, J., D'Mello, S. K., & Olney, A. (2015). Motion tracker: Camera-based monitoring of bodily movements using motion silhouettes. PloS ONE, 10(6). doi:10.1371/journal.pone.0130293
   Web of Science ®Google Scholar
• Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50(1), 537–567.
   PubMed Web of Science ®Google Scholar
• Larsen, J., Berntson, G., Poehlmann, K., Ito, T., & Cacioppo, J. (2008). The psychophysiology of emotion. In M. Lewis, J. Haviland-Jones, & L. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 180–195). New York, NY: Guilford.
   Google Scholar
• Lehman, B., Matthews, M., D'Mello, S. K., & Person, N. (2008). What are you feeling? Investigating student affective states during expert human tutoring sessions. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 50–59). Berlin, Germany: Springer.
   Google Scholar
• Lehr, C. A., Sinclair, M. F., & Christenson, S. L. (2004). Addressing student engagement and truancy prevention during the elementary school years: A replication study of the check & connect model. Journal of Education for Students Placed at Risk, 9, 279–301.
   Google Scholar
• Lewis, M. D. (2005). Bridging emotion theory and neurobiology through dynamic systems modeling. Behavioral and Brain Sciences, 28, 169–245.
   PubMed Web of Science ®Google Scholar
• Linnenbrink-Garcia, L., & Pekrun, R. (2011). Students' emotions and academic engagement: Introduction to the special issue. Contemporary Educational Psychology, 36(1), 1–3.
   Web of Science ®Google Scholar
• Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., & Bartlett, M. (2011). The computer expression recognition toolbox (CERT). In Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition (pp. 298–305). Washington, DC: IEEE.
   Google Scholar
• Loveless, T. (2015). How well are American students learning? Washington, DC: Brookings Institution.
   Google Scholar
• Mandler, G. (1990). Interruption (discrepancy) theory: Review and extensions. In S. Fisher & C. L. Cooper (Eds.), On the move: The psychology of change and transition (pp. 13–32). Chichester, UK: Wiley.
   Google Scholar
• Marshall, S. P. (2005). Assessing cognitive engagement and cognitive state from eye metrics. In D. D. Schmorrow (Ed.), Foundations of augmented cognition (pp. 312–320). Mahwah, NJ: Erlbaum.
   Google Scholar
• Martin, J. P. (2010). Lower Merion district's laptop saga ends with $610,000 settlement, The Philadelphia Inquirer. Retrieved from http://articles.philly.com/
   Google Scholar
• Matsumoto, D., Keltner, D., Shiota, M. N., O'Sullivan, M., & Frank, M. (2008). Facial expressions of emotion. In M. Lewis, J. Haviland-Jones, & L. Barrett (Eds.), Handbook of emotions (Vol. 3, pp. 211–234). New York, NY: Guilford Press.
   Google Scholar
• Mehl, M. R., Pennebaker, J. W., Crow, D. M., Dabbs, J., & Price, J. H. (2001). The Electronically Activated Recorder (EAR): A device for sampling naturalistic daily activities and conversations. Behavior Research Methods, Instruments, & Computers, 33, 517–523.
   PubMedGoogle Scholar
• Mehu, M., & Scherer, K. (2012). A psycho-ethological approach to social signal processing. Cognitive Processing, 13, 397–414.
   PubMed Web of Science ®Google Scholar
• Mesquita, B., & Boiger, M. (2014). Emotions in context: A sociodynamic model of emotions. Emotion Review, 6, 298–302.
   Web of Science ®Google Scholar
• Miller, B. W. (2015). Using reading times and eye-movements to measure cognitive engagement. Educational Psychologist, 50(1), 31–42.
   Web of Science ®Google Scholar
• Miller, W. L., Petsche, K., Baker, R. S., Labrum, M. J., & Wagner, A. Z. (2014). Boredom across activities, and across the year, within reasoning mind. Paper presented at the workshop on data mining for educational assessment and feedback (ASSESS 2014), New York, NY.
   Google Scholar
• Monkaresi, H., Bosch, N., Calvo, R. A., & D'Mello, S. K. (in press). Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Transactions on Affective Computing.
   Web of Science ®Google Scholar
• Mota, S., & Picard, R. (2003, June). Automated posture analysis for detecting learner's interest level. Paper presented at the Computer Vision and Pattern Recognition Workshop, Madison, WI.
   Google Scholar
• National Student Clearinghouse. (2016). http://www.studentclearinghouse.org/
   Google Scholar
• Niedenthal, P. M. (2007). Embodying emotion. Science, 316(5827), 1002–1005. http://dx.doi.org/10.1126/science.1136930
   PubMed Web of Science ®Google Scholar
• Nystrand, M., & Gamoran, A. (1991). Instructional discourse, student engagement, and literature achievement. Research in the Teaching of English, 25, 261–290.
   Web of Science ®Google Scholar
• Ocumpaugh, J., Baker, R., Gowda, S., Heffernan, N., & Heffernan, C. (2014). Population validity for educational data mining: A case study in affect detection. British Journal of Educational Psychology, 45, 487–501.
   Google Scholar
• Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T. (2012). Baker-Rodrigo Observation Method Protocol (BROMP) 1.0. Training Manual version 1.0. New York, NY: Teachers College, Columbia University.
   Google Scholar
• Ocumpaugh, J., Baker, R. S., & Rodrigo, M. M. T. (2015). Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 Technical and Training Manual. New York, NY and Manila, Philippines: Teachers College, Columbia University and Ateneo Laboratory for the Learning Sciences.
   Google Scholar
• OECD. (2012). PISA 2012 Results: Ready to Learn: Students' Engagement, Drive and Self-Beliefs (Vol. III). OECD Publishing.
   Google Scholar
• Olney, A., Risko, E. F., D'Mello, S. K., & Graesser, A. C. (2015). Attention in educational contexts: The role of the learning task in guiding attention. In J. Fawcett, E. F. Risko, & A. Kingstone (Eds.), The handbook of attention (pp. 623–642). Cambridge, MA: MIT Press.
   Google Scholar
• Pantic, M., & Patras, I. (2006). Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B., 36, 433–449. http://dx.doi.org/10.1109/tsmcb.2005.859075
   PubMed Web of Science ®Google Scholar
• Pardos, Z., Baker, R. S. J. D., San Pedro, M. O. C. Z., & Gowda, S. M. (2013). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the 3rd international conference on learning analytics and knowledge (pp. 117–124). New York, NY: ACM.
   Google Scholar
• Pekrun, R., Goetz, T., Daniels, L., Stupnisky, R. H., & Perry, R. (2010). Boredom in achievement settings: Exploring control–value antecedents and performance outcomes of a neglected emotion. Journal of Educational Psychology, 102, 531–549. http://dx.doi.org/10.1037/a0019243
   Web of Science ®Google Scholar
• Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259–282). New York, NY: Springer.
   Google Scholar
• Pianta, R. C., Hamre, B. K., & Allen, J. P. (2012). Teacher-student relationships and engagement: Conceptualizing, measuring, and improving the capacity of classroom interactions. In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 365–386). New York, NY: Springer.
   Google Scholar
• Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.
   PubMed Web of Science ®Google Scholar
• Poh, M. Z., McDuff, D. J., & Picard, R. W. (2011). Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1), 7–11.
   PubMed Web of Science ®Google Scholar
• Porayska-Pomsta, K., Mavrikis, M., D'Mello, S. K., Conati, C., & Baker, R. (2013). Knowledge elicitation methods for affect modelling in education. International Journal of Artificial Intelligence in Education, 22, 107–140.
   Google Scholar
• Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422.
   PubMed Web of Science ®Google Scholar
• Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N. T., Koedinger, K. R., Junker, B., & Choksey, S. (2005). The Assistment project: Blending assessment and assisting. In C. Loi & G. McCalla (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 555–562). Amsterdam, The Netherlands: IOS Press.
   Google Scholar
• Reeve, J., & Tseng, C. M. (2011). Agency as a fourth aspect of students' engagement during learning activities. Contemporary Educational Psychology, 36, 257–267.
   Web of Science ®Google Scholar
• Renninger, K. A., & Bachrach, J. E. (2015). Studying triggers for interest and engagement using observational methods. Educational Psychologist, 50(1), 58–69.
   Web of Science ®Google Scholar
• Reschly, A., & Christenson, S. (2012). Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. In S. Christenson, A. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 3–19). Berlin, Germany: Springer.
   Google Scholar
• Roseman, I. J. (2011). Emotional behaviors, emotivational goals, emotion strategies: Multiple levels of organization integrate variable and consistent responses. Emotion Review, 3, 434–443.
   Web of Science ®Google Scholar
• Ruch, W. (1995). Will the real relationship between facial expression and affective experience please stand up: The case of exhilaration. Cognition & Emotion, 9(1), 33–58.
   Web of Science ®Google Scholar
• Russell, J. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172.
   PubMed Web of Science ®Google Scholar
• Russell, J. A., Bachorowski, J. A., & Fernandez-Dols, J. M. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329–349.
   PubMed Web of Science ®Google Scholar
• Ryan, R., & Deci, E. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78.
   PubMed Web of Science ®Google Scholar
• Ryu, S., & Lombardi, D. (2015). Coding classroom interactions for collective and individual engagement. Educational Psychologist, 50(1), 70–83.
   Web of Science ®Google Scholar
• Sabourin, J., Mott, B., & Lester, J. (2011). Modeling learner affect with theoretically grounded dynamic bayesian networks. In S. D'Mello, A. Graesser, B. Schuller, & J. Martin (Eds.), Proceedings of the Fourth International Conference on Affective Computing and Intelligent Interaction (pp. 286–295). Berlin, Germany: Springer-Verlag.
   Google Scholar
• San Pedro, M., Baker, R. S., Bowers, A. J., & Heffernan, N. T. (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In S. D'Mello, R. Calvo, & A. Olney (Eds.), Proceedings of the 6th international conference on educational data mining (EDM 2013) (pp. 177–184). Retrieved from http://www.educationaldatamining.org/conferences/index.php/EDM/2013/paper/view/1035/1001
   Google Scholar
• Schunk, D. H., & Pajares, F. (2005). Competence perceptions and academic functioning. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 141–163). New York, NY: Guilford.
   Google Scholar
• Scully, C. G., Lee, J., Meyer, J., Gorbach, A. M., Granquist-Fraser, D., Mendelson, Y., & Chon, K. H. (2012). Physiological parameter monitoring from optical recordings with a mobile phone. IEEE Transactions on Biomedical Engineering, 59, 303–306.
   PubMed Web of Science ®Google Scholar
• Sewell, W., & Komogortsev, O. (2010). Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network. In Proceedings of the 2012 ACM annual conference on human factors in computing systems (pp. 3739–3744). New York, NY: ACM.
   Google Scholar
• Shibata, K., Watanabe, T., Sasaki, Y., & Kawato, M. (2011). Perceptual learning incepted by decoded fMRI neurofeedback without stimulus presentation. Science, 334(6061), 1413–1415.
   PubMed Web of Science ®Google Scholar
• Shute, V. J., Ventura, M., & Kim, Y. J. (2013). Assessment and learning of qualitative physics in Newton's playground. The Journal of Educational Research, 106, 423–430.
   Web of Science ®Google Scholar
• Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and measuring student engagement in science. Educational Psychologist, 50(1), 1–13.
   Web of Science ®Google Scholar
• Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology, 85(4), 571–581.
   Web of Science ®Google Scholar
• Smallwood, J., McSpadden, M., & Schooler, J. W. (2008). When attention matters: The curious incident of the wandering mind. Memory & Cognition, 36, 1144–1150. http://dx.doi.org/10.3758/mc.36.6.1144
   PubMed Web of Science ®Google Scholar
• St. John, M., Kobus, D. A., Morrison, J. G., & Schmorrow, D. (2004). Overview of the DARPA augmented cognition technical integration experiment. International Journal of Human–Computer Interaction, 17, 131–149.
   Web of Science ®Google Scholar
• Stein, N., & Levine, L. (1991). Making sense out of emotion. In W. Kessen, A. Ortony, & F. Kraik (Eds.), Memories, thoughts, and emotions: Essays in honor of George Mandler (pp. 295–322). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Turner, J. C., & Meyer, D. K. (2000). Studying and understanding the instructional contexts of classrooms: Using our past to forge our future. Educational Psychologist, 35(2), 69–85.
   Web of Science ®Google Scholar
• Valstar, M., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012). Meta-analysis of the first facial expression recognition challenge. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42, 966–979.
   PubMed Web of Science ®Google Scholar
• Volpe, R. J., DiPerna, J. C., Hintze, J. M., & Shapiro, E. S. (2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review, 34, 454–474.
   Web of Science ®Google Scholar
• Wachowski Brothers. (1999). The matrix [Motion picture]. United States: Warner Brothers.
   Google Scholar
• Wang, R., Chen, F., Chen, Z., Li, T., Harari, G., Tignor, S., & Campbell, A. T. (2014). StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In J. Kientz, J. Scott, & J. Song (Eds.), Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (ACM UbiComp ′14) (pp. 3–14). New York, NY: ACM.
   Google Scholar
• Whitehill, J., Serpell, Z., Lin, Y. C., Foster, A., & Movellan, J. (2014). The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing, 5(1), 86–98.
   Web of Science ®Google Scholar