Analysing student engagement with 360-degree videos through multimodal data analytics and user annotations

References

• Alzahrani, A., Gardner, M., Callaghan, V., & Alrashidi, M. (2015). Towards measuring learning effectiveness considering presence, engagement and immersion in a mixed and augmented reality learning environment. In D. Preuveneers (Ed.), Ebook: Workshop proceedings of the 11th International Conference on Intelligent Environments (pp. 252–264). IOS Press. https://doi.org/10.3233/978-1-61499-530-2-252
   Google Scholar
• Andrade, A., Delandshere, G., & Danish, J. A. (2016). Using multimodal learning analytics to model student behavior: a systematic analysis of epistemological framing. Journal of Learning Analytics, 3(2), 282–306. https://doi.org/10.18608/jla.2016.32.14
   Google Scholar
• Argyriou, L., Economou, D., & Bouki, V. (2017). 360 degree interactive video application for cultural heritage education. In D. Beck, C. Allison, L. Morgado, J. Pirker, F. Khosmood, J. Richter, & C. Gütl (Eds.), iLRN 2017: Coimbra workshop, long and short paper, and poster proceedings from the Third Immersive Learning Research Network Conference (pp. 297–304). Verlag der Technischen Universität Graz. https://doi.org/10.3217/978-3-85125-609-3
   Google Scholar
• Assilmia, F., Yun, S. P., Okawa, K., & Kunze, K. (2017). IN360: A 360 degree-video platform to change students preconceived notions on their career. In CHI Conference Extended abstracts on Human Factors in Computing Systems (pp. 2359–2365). ACM. https://doi.org/10.1145/3027063.3053211
   Google Scholar
• Bernstein, B. (1990). Class, codes and control: The structuring of pedagogic discourse (Vol. 4). Routledge.
   Google Scholar
• Bernstein, B. (2000). Pedagogy, symbolic control and identity: Theory, research, critique. Taylor & Francis.
   Google Scholar
• Blikstad-Balas, M. (2017). Key challenges of using video when investigating social practices in education: Contextualization, magnification, and representation. International Journal of Research & Method in Education, 40(5), 511–523. https://doi.org/10.1080/1743727X.2016.1181162
   Web of Science ®Google Scholar
• Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://doi.org/10.18608/jla.2016.32.11
   Google Scholar
• Boling, E. C., Hough, M., Krinsky, H., Saleem, H., & Stevens, M. (2012). Cutting the distance in distance education: Perspectives on what promotes positive, online learning experiences. The Internet and Higher Education, 15(2), 118–126. https://doi.org/10.1016/j.iheduc.2011.11.006
   Web of Science ®Google Scholar
• Chittleborough, G., Cripps Clark, J., & Chandler, P. (2015). The pedagogy of using video to develop reflective practice in learning to teach science. In E. Ortlieb, L. E. Shanahan, & M. B. McVee (Eds.), Video research in disciplinary literacies (pp. 95–115). Emerald Group Publishing.
   Google Scholar
• Christie, F. (2005). Classroom discourse analysis: A functional perspective. Continuum.
   Google Scholar
• Ciekanski, M., & Chanier, T. (2008). Developing online multimodal verbal communication to enhance the writing process in an audio-graphic conferencing environment. ReCALL, 20(2), 162–182. https://doi.org/10.1017/S0958344008000426
   Google Scholar
• Crook, A., Mauchline, A., Maw, S., Lawson, C., Drinkwater, R., Lundqvist, K., Orsmond, P., Gomez, S., & Park, J. (2012). The use of video technology for providing feedback to students: Can it enhance the feedback experience for staff and students? Computers & Education, 58(1), 386–396. https://doi.org/10.1016/j.compedu.2011.08.025
   Web of Science ®Google Scholar
• Davidsen, J., & Vanderlinde, R. (2014). Researchers and teachers learning together and from each other using video-based multimodal analysis. British Journal of Educational Technology, 45(3), 451–460. https://doi.org/10.1111/bjet.12141
   Web of Science ®Google Scholar
• Eggins, S. (2005). An introduction to systemic functional linguistics (2nd ed.). Pinter Publishers.
   Google Scholar
• Elmezeny, A., Edenhofer, N., & Wimmer, J. (2018). Immersive storytelling in 360-degree videos: An analysis of interplay between narrative and technical immersion. Journal for Virtual Worlds Research, 11(1), 1–13. https://doi.org/10.4101/jvwr.v11i1.7298
   Google Scholar
• Georgiou, Y., & Kyza, E. A. (2017). A design-based approach to augmented reality location-based activities: Investigating immersion in relation to student learning. In F. Loizides, G. Papadopoulos, & N. Souleles (Eds.), mLearn 2017: Proceedings of the 16th World Conference on Mobile and Contextual Learning (pp. 1–8). Association for Computing Machinery. https://doi.org/10.1145/3136907.3136926
   Google Scholar
• Gill, P., & Remedios, R. (2013). How should researchers in Education operationalise on-task behaviours? Cambridge Journal of Education, 43(2), 199–222. https://doi.org/10.1080/0305764X.2013.767878
   Web of Science ®Google Scholar
• Goodwin, C., & Duranti, A. (1992). Rethinking context: An introduction. In C. Goodwin & A. Duranti (Eds.), Rethinking context: Language as an interactive phenomenon(pp. 1–42). Cambridge University Press.
   Google Scholar
• Gregory, S., Lee, M. J., Dalgarno, B., & Tynan, B. (2016). Learning in virtual worlds: Research and applications. Athabasca University Press.
   Google Scholar
• Hackling, M. W., Murcia, K. J., Ibrahim-Didi, K., & Hill, S. M. (2014). Methods for multimodal analysis and representation of teaching-learning interactions in primary science lessons captured on video. In C. P. Constantinou, N. Papadouris, & A. Hadjigeorgiou (Eds.), Proceedings of the ESERA 2013 Conference: Science Education Research for Evidence-based Teaching and Coherence in Learning (pp. 1–8). European Science Education Research Association. https://www.esera.org/publications/esera-conference-proceedings/esera-2013#strand-3
   Google Scholar
• Hales, P., & Kalyvaki, M. (2017). Immersive learning experiences in teacher education. In P. Resta & S. Smith (Eds.), Society for Information Technology & Teacher Education International Conference (pp. 1586–1587). Association for the Advancement of Computing in Education. https://www.learntechlib.org/primary/p/177439/
   Google Scholar
• Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. Edward Arnold.
   Google Scholar
• Halliday, M. A. K. (1994). An introduction to functional grammar (2nd ed.). Arnold.
   Google Scholar
• Halliday, M. A. K., & Matthiessen, C. M. I. M. (2014). Halliday’s introduction to functional grammar. Routledge.
   Google Scholar
• Healion, D., Russell, S., Cukurova, M., & Spikol, D. (2017). Tracing physical movement during practice-based learning through multimodal learning analytics. In LAK ’17: Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 588–589). ACM. https://doi.org/10.1145/3027385.3029474
   Google Scholar
• Hung, I. C., & Chen, N. S. (2018). Embodied interactive video lectures for improving learning comprehension and retention. Computers & Education, 117, 116–131. https://doi.org/10.1016/j.compedu.2017.10.005
   Web of Science ®Google Scholar
• Hung, S.-T. A. (2016). Enhancing feedback provision through multimodal video technology. Computers & Education, 98, 90–101. https://doi.org/10.1016/j.compedu.2016.03.009
   Web of Science ®Google Scholar
• Jewitt, C. (2003). Re-thinking assessment: Multimodality, literacy and computer-mediated learning. Assessment in Education: Principles, Policy & Practice, 10(1), 83–102. https://doi.org/10.1080/09695940301698
   Google Scholar
• Jewitt, C. (2012). An introduction to using video for research. National Centre for Research Methods Working Paper 03/12. http://eprints.ncrm.ac.uk/2259/4/NCRM_workingpaper_0312.pdf
   Google Scholar
• Jewitt, C., Kress, G., Ogborn, J., & Tsatsarelis, C. (2001). Exploring learning through visual, actional and linguistic communication: The multimodal environment of a science classroom. Educational Review, 53(1), 5–18. https://doi.org/10.1080/00131910123753
   Web of Science ®Google Scholar
• Kessler, K., & Thomson, L. A. (2010). The embodied nature of spatial perspective taking: Embodied transformation versus sensorimotor interference. Cognition, 114(1), 72–88. https://doi.org/10.1016/j.cognition.2009.08.015
   PubMed Web of Science ®Google Scholar
• Leijon, M. (2016). Space as designs for and in learning: Investigating the interplay between space, interaction and learning sequences in higher education. Visual Communication, 15(1), 93–124. https://doi.org/10.1177/1470357215608553
   Web of Science ®Google Scholar
• Lim, F. V., O’Halloran, K. L., & Podlasov, A. (2012). Spatial pedagogy: Mapping meanings in the use of classroom space. Cambridge Journal of Education, 42(2), 235–251. https://doi.org/10.1080/0305764X.2012.676629
   Web of Science ®Google Scholar
• Major, L., & Watson, S. (2018). Using video to support in-service teacher professional development: The state of the field, limitations and possibilities. Technology, Pedagogy and Education, 27(1), 49–68. https://doi.org/10.1080/1475939X.2017.1361469
   Web of Science ®Google Scholar
• Martin, J. R. (2002). Meaning beyond the clause: SFL perspectives. Annual Review of Applied Linguistics, 22, 52–74. https://doi.org/10.1017/S026719050200003X
   Google Scholar
• Martin, J. R., & Rose, D. (2007). Working with discourse: Meaning beyond the clause (2nd ed.). Continuum.
   Google Scholar
• Martin, J. R., & White, P. R. R. (2005). The language of evaluation: Appraisal in English. Palgrave Macmillan.
   Google Scholar
• McGowan, A., & Hanna, P. (2015). How video lecture capture affects student engagement in a higher education computer programming course: A study of attendance, video viewing behaviours and student attitude. In eChallenges e-2015 Conference (pp. 1–8). IEEE. https://doi.org/10.1109/eCHALLENGES.2015.7440966
   Google Scholar
• Morell, T. (2015). International conference paper presentations: Amultimodal analysis to determine effectiveness. English for Specific Purposes, 37, 137–150. https://doi.org/10.1016/j.esp.2014.10.002
   Web of Science ®Google Scholar
• Morell, T. (2017). Multimodal competence and effective interactive lecturing. System, 77, 70–79. https://doi.org/10.1016/j.system.2017.12.006
   Web of Science ®Google Scholar
• O’Halloran, K. L. (2011). The semantic hyperspace: Accumulating mathematical knowledge across semiotic resources and modes. In F. Christie & K. Maton (Eds.), Disciplinarity: Functional linguistic and sociological perspectives (pp. 217–236).
   Google Scholar
• O’Halloran, K. L. (2015). The language of learning mathematics: Amultimodal perspective. The Journal of Mathematical Behaviour, 40(Part A), 63–74. https://doi.org/10.1016/j.jmathb.2014.09.002
   Google Scholar
• O’Halloran, K. L., Tan, S., Wiebrands, M., Sheffield, R., Wignell, P., & Turner, P. (2018). The multimodal classroom in the digital age: The use of 360 degree videos for online learning. In H. de Silva Joyce & S. Feez (Eds.), Multimodality across classrooms: Learning about and through different modalities.
   Google Scholar
• Owens, A., & Efros, A. A. (2018). Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 631–648). Computer Vision Foundation. https://openaccess.thecvf.com/content_ECCV_2018/html/Andrew_Owens_Audio-Visual_Scene_Analysis_ECCV_2018_paper.html
   Google Scholar
• Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez-Triana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193–203. https://doi.org/10.1111/jcal.12232
   PubMed Web of Science ®Google Scholar
• Rose, D. (2014). Analysing pedagogic discourse: An approach from genre and register. Functional Linguistics, 1(11), 1–32. https://doi.org/10.1186/s40554-014-0011-4
   Google Scholar
• Ryan, J., Scott, A., & Walsh, M. (2010). Pedagogy in the multimodal classroom: An analysis of the challenges and opportunities for teachers. Teachers and Teaching: Theory and Practice, 16(4), 477–489. https://doi.org/10.1080/13540601003754871
   Web of Science ®Google Scholar
• Sakr, M., Jewitt, C., & Price, S. (2016). Mobile experiences of historical place: Amultimodal analysis of emotional engagement. Journal of the Learning Sciences, 25(1), 51–92. https://doi.org/10.1080/10508406.2015.1115761
   Web of Science ®Google Scholar
• Salselas, I., & Penha, R. (2019). The role of sound in inducing storytelling in immersive environments. In AM ’19: Proceedings of the 14th International Audio Mostly Conference: A Journey in Sound on ZZZ (pp. 191–198). ACM. https://doi.org/10.1145/3356590.3356619
   Google Scholar
• Sarker, B. (2017). Decoding the user experience in mobile virtual reality narratives. In S. Lackey & J. Chen (Eds.), International Conference on Virtual, Augmented and Mixed Reality (pp. 437–452). Springer. https://doi.org/10.1007/978-3-319-57987-0_36
   Google Scholar
• Sauli, F., Cattaneo, A., & van der Meij, H. (2018). Hypervideo for educational purposes: A literature review on a multifaceted technological tool. Technology, Pedagogy and Education, 27(1), 115–134. https://doi.org/10.1080/1475939X.2017.1407357
   Web of Science ®Google Scholar
• Sheikh, A., Brown, A., Watson, Z., & Evans, M. (2016). Directing attention in 360-degree video. In IBC 2016 Conference. https://doi.org/10.1049/ibc.2016.0029
   Google Scholar
• Shernoff, D. J. (2013). Optimal learning environments to promote student engagement. Springer.
   Google Scholar
• Smith, C., King, B., & Gonzalez, D. (2016). Using multimodal learning analytics to identify patterns of interactions in a body-based mathematics activity. Journal of Interactive Learning Research, 27(4), 355–379. https://www.learntechlib.org/primary/p/151124/
   Google Scholar
• Sümer, O., Goldberg, P., Stürmer, K., Seidel, T., Gerjets, P., Trautwein, U., & Kasneci, E. (2018). Teachers’ perception in the classroom. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (pp. 2315–2324). Computer Vision Foundation. https://openaccess.thecvf.com/content_cvpr_2018_workshops/w47/html/Sumer_Teachers_Perception_in_CVPR_2018_paper.html
   Google Scholar
• Tan, S., O’Halloran, K. L., & Wignell, P. (2016). Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL. ReCALL, 28(3), 253–273. https://doi.org/10.1017/S0958344016000124
   Web of Science ®Google Scholar
• Twiner, A., Coffin, C., Littleton, K., & Whitelock, D. (2010). Multimodality, orchestration and participation in the context of classroom use of the interactive whiteboard: A discussion. Technology, Pedagogy and Education, 19(2), 211–223. https://doi.org/10.1080/1475939X.2010.491232
   Web of Science ®Google Scholar
• van Es, E. A., Cashen, M., Barnhart, T., & Auger, A. (2017). Learning to notice mathematics instruction: Using video to develop preservice teachers’ vision of ambitious pedagogy. Cognition and Instruction, 35(3), 165–187. https://doi.org/10.1080/07370008.2017.1317125
   Web of Science ®Google Scholar
• van Es, E. A., Stockero, S. L., Sherin, M. G., Van Zoest, L. R., & Dyer, E. (2015). Making the most of teacher self-captured video. Mathematics Teacher Educator, 4(1), 6–19. https://doi.org/10.5951/mathteaceduc.4.1.0006
   Google Scholar
• Wang, J., & Antonenko, P. D. (2017). Instructor presence in instructional video: Effects on visual attention, recall, and perceived learning. Computers in Human Behavior, 71, 79–89. https://doi.org/10.1016/j.chb.2017.01.049
   Web of Science ®Google Scholar
• Wang, X., Lin, L., Han, M., & Spector, J. M. (2020). Impacts of cues on learning: using eye-tracking technologies to examine the functions and designs of added cues in short instructional videos. Computers in Human Behavior, 107, 106279. https://doi.org/10.1016/j.chb.2020.106279
   Web of Science ®Google Scholar
• Zappavigna, M., & Zhao, S. (2017). Selfies in ‘mommyblogging’: An emerging visual genre. Discourse, Context & Media, 20, 239–247. https://doi.org/10.1016/j.dcm.2017.05.005
   Web of Science ®Google Scholar