Emotional Experience during Human-Computer Interaction: A Survey

References

• Baker, R. S., D’Mello, S. K., Rodrigo, M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003
   Web of Science ®Google Scholar
• Barrett, L. F. (2014). The conceptual act theory: A précis. Emotion Review, 6(4), 292–297. https://doi.org/10.1177/1754073914534479
   Web of Science ®Google Scholar
• Barrett, L. F., Wilson-Mendenhall, C. D., & Barsalou, L. W. (2015). The conceptual act theory: A roadmap. In L. F. Barrett & J. A. Russell (Eds.), The psychological construction of emotion (pp. 83–110). The Guilford Press.
   Google Scholar
• Bedny, G. Z., & Karwowski, W. (2011). Application of systemic-structural theory of activity. In W. W. Gasparski & T. Airaksinen (Eds.), Praxiology and the philosophy of technology (pp. 97–144). Transaction Publishers.
   Google Scholar
• Braun, M., Weber, F., & Alt, F. (2022). Affective automotive user interfaces–reviewing the state of driver affect research and emotion regulation in the car. ACM Computing Surveys, 54(7), 1–26. https://doi.org/10.1145/3460938
   Web of Science ®Google Scholar
• Brave, S., & Nass, C. (2007). Emotion in human-computer interaction. In The human-computer interaction handbook (pp. 103–118). CRC Press. https://doi.org/10.1201/9781410615862
   Google Scholar
• Camodeca, M., & Nava, E. (2022). The long-term effects of bullying, victimization, and bystander behavior on emotion regulation and its physiological correlates. Journal of Interpersonal Violence, 37(3–4), NP2056–NP2075. https://doi.org/10.1177/0886260520934438
   PubMed Web of Science ®Google Scholar
• Chuah, S. H. W., & Yu, J. (2021). The future of service: The power of emotion in human-robot interaction. Journal of Retailing and Consumer Services, 61, 102551. https://doi.org/10.1016/j.jretconser.2021.102551
   Web of Science ®Google Scholar
• Colnerič, N., & Demšar, J. (2018). Emotion recognition on twitter: Comparative study and training a unison model. IEEE Transactions on Affective Computing, 11(3), 433–446. https://doi.org/10.1109/TAFFC.2018.2807817
   Web of Science ®Google Scholar
• Cowen, A. S., & Keltner, D. (2017). Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences of the United States of America, 114(38), E7900–E7909. https://doi.org/10.1073/pnas.1702247114
   PubMed Web of Science ®Google Scholar
• Cowen, A. S., & Keltner, D. (2020). Universal facial expressions uncovered in art of the ancient Americas: A computational approach. Science Advances, 6(34), eabb1005. https://doi.org/10.1126/sciadv.abb1005
   PubMed Web of Science ®Google Scholar
• Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001). Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine, 18(1), 32–80. https://doi.org/10.1109/79.911197
   Web of Science ®Google Scholar
• Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250. https://doi.org/10.1080/1358165042000283101
   Google Scholar
• Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. Harper Collins.
   Google Scholar
• D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082. https://doi.org/10.1037/a0032674
   Web of Science ®Google Scholar
• Devillers, L., Tahon, M., Sehili, M. A., & Delaborde, A. (2015). Inference of human beings’ emotional states from speech in human–robot interactions. International Journal of Social Robotics, 7(4), 451–463. https://doi.org/10.1007/s12369-015-0297-8
   Web of Science ®Google Scholar
• Dunlop, M., & Brewster, S. (2002). The challenges of mobile devices for human computer interaction. Personal and Ubiquitous Computing, 6(4), 235–236. https://doi.org/10.1007/s007790200022
   Web of Science ®Google Scholar
• Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3–4), 169–200. https://doi.org/10.1080/02699939208411068
   Web of Science ®Google Scholar
• Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologist Press. https://doi.org/10.1037/t27734-000
   Google Scholar
• Ellsworth, P. C., & Smith, C. A. (1988). From appraisal to emotion: Differences among unpleasant feelings. Motivation and Emotion, 12(3), 271–302. https://doi.org/10.1007/BF00993115
   Web of Science ®Google Scholar
• Fabri, M., Elzouki, S. Y. A., & Moore, D. (2007). Emotionally expressive avatars for chatting, learning and therapeutic intervention. In Human-Computer Interaction. HCI Intelligent Multimodal Interaction Environments: 12th International Conference, HCI International 2007, Beijing, China, July 22-27, 2007, Proceedings, Part III 12 (pp. 275–285). Springer Berlin Heidelberg.
   Google Scholar
• Fragopanagos, N., & Taylor, J. G. (2005). Emotion recognition in human–computer interaction. Neural Networks, 18(4), 389–405. https://doi.org/10.1016/j.neunet.2005.03.006
   PubMed Web of Science ®Google Scholar
• Franklin, S. B., Gibson, D. J., Robertson, P. A., Pohlmann, J. T., & Fralish, J. S. (1995). Parallel analysis: A method for determining significant principal components. Journal of Vegetation Science, 6(1), 99–106. https://doi.org/10.2307/3236261
   Web of Science ®Google Scholar
• Giakoumis, D., Vogiannou, A., Kosunen, I., Moustakas, K., Tzovaras, D., & Hassapis, G. (2010, January). Identifying psychophysiological correlates of boredom and negative mood induced during HCI. In B-Interface, pp. 3–12.
   Google Scholar
• Goldenberg, A., & Willer, R. (2023). Amplification of emotion on social media. Nature Human Behaviour, 7(6), 845–846. https://doi.org/10.1038/s41562-023-01604-x
   PubMed Web of Science ®Google Scholar
• Graesser, A., D’Mello, S., Chipman, P., King, B., & McDaniel, B. (2007). Exploring relationships between affect and learning with AutoTutor. In Proc Int Conf AIED.
   Google Scholar
• Hamann, S. (2012). Mapping discrete and dimensional emotions onto the brain: Controversies and consensus. Trends in Cognitive Sciences, 16(9), 458–466. https://doi.org/10.1016/j.tics.2012.07.006
   PubMed Web of Science ®Google Scholar
• Hareli, S., & Parkinson, B. (2008). What’s social about social emotions? Journal for the Theory of Social Behaviour, 38(2), 131–156. https://doi.org/10.1111/j.1468-5914.2008.00363.x
   Web of Science ®Google Scholar
• Harrison, S., Tatar, D., & Sengers, P. (2007, April). The three paradigms of HCI. In Alt. Chi. Session at the SIGCHI Conference on human factors in computing systems San Jose (pp. 1–18).
   Google Scholar
• Hasnine, M. N., Bui, H. T., Tran, T. T. T., Nguyen, H. T., Akçapınar, G., & Ueda, H. (2021). Students’ emotion extraction and visualization for engagement detection in online learning. Procedia Computer Science, 192, 3423–3431. https://doi.org/10.1016/j.procs.2021.09.115
   Google Scholar
• Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191–205. https://doi.org/10.1177/1094428104263675
   Web of Science ®Google Scholar
• Hibbeln, M., Jenkins, J. L., Schneider, C., Valacich, J. S., & Weinmann, M. (2017). How is your user feeling? Inferring emotion through human–computer interaction devices. MIS Quarterly, 41(1), 1–21. https://doi.org/10.25300/MISQ/2017/41.1.01
   Web of Science ®Google Scholar
• Hinkle, L. B., Roudposhti, K. K., & Metsis, V. (2019, June). Physiological measurement for emotion recognition in virtual reality. In 2019 2nd International Conference on Data Intelligence and Security (ICDIS)) (pp. 136–143). IEEE. https://doi.org/10.1109/ICDIS.2019.00028
   Google Scholar
• Hu, X., Wang, F., & Zhang, D. (2022). Similar brains blend emotion in similar ways: Neural representations of individual difference in emotion profiles. NeuroImage, 247, 118819. https://doi.org/10.1016/j.neuroimage.2021.118819
   PubMed Web of Science ®Google Scholar
• Huang, K. Y. (2009). Challenges in human-computer interaction design for mobile devices. In Proceedings of the World Congress on Engineering and Computer Science (Vol. 1, No. 6, pp. 1–6).
   Google Scholar
• Jerritta, S., Murugappan, M., Nagarajan, R., & Wan, K. (2011, March). Physiological signals based human emotion recognition: A review. In 2011 IEEE 7th International Colloquium on Signal Processing and Its Applications) (pp. 410–415). IEEE. https://doi.org/10.1109/CSPA.2011.5759912
   Google Scholar
• Jokinen, J. P., & Silvennoinen, J. (2020). The appraisal theory of emotion in human–computer interaction. In Emotions in technology design: From experience to ethics (pp. 27–39). Springer International Publishing. https://doi.org/10.1007/978-3-030-53483-7_3
   Google Scholar
• Jones, S. M., McGarrah, M. W., & Kahn, J. (2019). Social and emotional learning: A principled science of human development in context. Educational Psychologist, 54(3), 129–143. https://doi.org/10.1080/00461520.2019.1625776
   Web of Science ®Google Scholar
• Khalil, R. A., Jones, E., Babar, M. I., Jan, T., Zafar, M. H., & Alhussain, T. (2019). Speech emotion recognition using deep learning techniques: A review. IEEE Access. 7, 117327–117345. https://doi.org/10.1109/ACCESS.2019.2936124
   Google Scholar
• Kort, B., Reilly, R., Picard, R. W. (2001). An affective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion. In Proceedings IEEE International Conference on Advanced Learning Technologies (pp. 43–46). IEEE. https://doi.org/10.1109/ICALT.2001.943850
   Google Scholar
• Kuutti, K. (1996). Activity theory as a potential framework for human-computer interaction research. In B. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 17–44). MIT Press.
   Google Scholar
• Kuutti, K., Bannon, L. J. (2014). The turn to practice in HCI: Towards a research agenda. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3543–3552). https://doi.org/10.1145/2556288.2557111
   Google Scholar
• Lang, P. J. (1984). Cognition in emotion: concept and action. In C. E. Izard, J. Kagan, & R. B. Zajonc (Eds.), Emotions, cognition, and behavior (pp. 192–228). Cambridge University Press.
   Google Scholar
• Leont’ev, A. N. (1978). Activity, consciousness, and personality. Prentice-Hall.
   Google Scholar
• Liu, H., Wang, F., & Zhang, D. (2023). Inspiring real-time evaluation and optimization of human–Robot interaction with psychological findings from human–Human interaction. Applied Sciences, 13(2), 676. https://doi.org/10.3390/app13020676
   Google Scholar
• Ludema, J. D., Wilmot, T. B., & Srivastva, S. (1997). Organizational hope: Reaffirming the constructive task of social and organizational inquiry. Human Relations, 50(8), 1015–1052. https://doi.org/10.1023/A:1016985129104
   Web of Science ®Google Scholar
• Luo, M., Hsu, T. W., Park, J. S., & Hancock, J. T. (2020). Emotional amplification during live-streaming: Evidence from comments during and after news events. Proceedings of the ACM on Human, 4(CSCW1), 1–19. https://doi.org/10.1145/3392853
   Google Scholar
• Makelberge, N. (2004). Flow, interaction design and contemporary boredom. IT University of Gothenburg.
   Google Scholar
• Martinent, G., Campo, M., & Ferrand, C. (2012). A descriptive study of emotional process during competition: Nature, frequency, direction, duration and co-occurrence of discrete emotions. Psychology of Sport and Exercise, 13(2), 142–151. https://doi.org/10.1016/j.psychsport.2011.10.006
   Web of Science ®Google Scholar
• Parasuraman, S., & Purohit, Y. S. (2000). Distress and boredom among orchestra musicians: The two faces of stress. Journal of Occupational Health Psychology, 5(1), 74–83. https://doi.org/10.1037/1076-8998.5.1.74
   PubMedGoogle Scholar
• Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In Handbook of research on student engagement (pp. 259–282). https://doi.org/10.1007/978-1-4614-2018-7_12
   Google Scholar
• Pérez-Espinosa, H., Zatarain-Cabada, R., & Barrón-Estrada, M. L. (2022). Emotion recognition: From speech and facial expressions. In Biosignal processing and classification using computational learning and intelligence (pp. 307–326). Academic Press. https://doi.org/10.1016/B978-0-12-820125-1.00028-2
   Google Scholar
• Piana, S., Stagliano, A., Odone, F., Verri, A., & Camurri, A. (2014). Real-time automatic emotion recognition from body gestures. arXiv Preprint arXiv:1402.5047. https://doi.org/10.48550/arXiv.1402.5047
   Google Scholar
• Picard, R. W. (2003). Computers that recognize and respond to user emotion. In International conference on user modeling (pp. 1–2). Springer. https://doi.org/10.1007/3-540-44963-9_2
   Google Scholar
• Picard, R. W. (2010). Emotion research by the people, for the people. Emotion Review, 2(3), 250–254. https://doi.org/10.1177/1754073910364256
   Web of Science ®Google Scholar
• Ramakrishnan, S., & El Emary, I. M. (2013). Speech emotion recognition approaches in human computer interaction. Telecommunication Systems, 52(3), 1467–1478. https://doi.org/10.1007/s11235-011-9624-z
   Web of Science ®Google Scholar
• Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/h0077714
   Web of Science ®Google Scholar
• Sreeja, P. S., & Mahalakshmi, G. (2017). Emotion models: A review. International Journal of Control Theory and Applications, 10(8), 651–657.
   Google Scholar
• Sawyer, K. B., & Clair, J. A. (2022). Hope cultures in organizations: Tackling the grand challenge of commercial sex exploitation. Administrative Science Quarterly, 67(2), 289–338. https://doi.org/10.1177/00018392211055506
   Web of Science ®Google Scholar
• Sensor Tower (2022). The State of Travel Apps 2022. Sensor Tower.
   Google Scholar
• Sensor Tower. (2023). 2023 Global Mobile Application (Non-Game) Market Outlook. Sensor Tower.
   Google Scholar
• Shechtman, N., Horowitz, L. M. (2003, April). Media inequality in conversation: How people behave differently when interacting with computers and people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 281–288). https://doi.org/10.1145/642611.642661
   Google Scholar
• Shen, H. Y., & Wang, M. (2022). Improving interaction experience through lexical convergence: The prosocial effect of lexical alignment in human-human and human-computer interactions. International Journal of Human–Computer Interaction, 38(1), 28–41. https://doi.org/10.1080/10447318.2021.1921367
   Web of Science ®Google Scholar
• Shui, X., Zhang, M., Li, Z., Hu, X., Wang, F., & Zhang, D. (2021). A dataset of daily ambulatory psychological and physiological recording for emotion research. Scientific Data, 8(1), 161. https://doi.org/10.1038/s41597-021-00945-4
   PubMed Web of Science ®Google Scholar
• Siegert, I., Bock, R., Wendemuth, A., Vlasenko, B., & Ohnemus, K. (2015, October). Overlapping speech, utterance duration and affective content in HHI and HCI-An comparison. In 2015 6th IEEE International Conference on Cognitive Infocommunications (CogInfoCom) (pp. 83–88). IEEE. https://doi.org/10.1109/CogInfoCom.2015.7390569
   Google Scholar
• Song, T., Zheng, W., Lu, C., Zong, Y., Zhang, X., & Cui, Z. (2019). MPED: A multi-modal physiological emotion database for discrete emotion recognition. IEEE Access. 7, 12177–12191. https://doi.org/10.1109/ACCESS.2019.2891579
   Web of Science ®Google Scholar
• Stock-Homburg, R. (2022). Survey of Emotions in Human–Robot Interactions: Perspectives from Robotic Psychology on 20 Years of Research. International Journal of Social Robotics, 14(2), 389–411. https://doi.org/10.1007/s12369-021-00778-6
   Web of Science ®Google Scholar
• Tyng, C. M., Amin, H. U., Saad, M. N., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8, 1454. https://doi.org/10.3389/fpsyg.2017.01454
   PubMed Web of Science ®Google Scholar
• Uden, L. (2007). Activity theory for designing mobile learning. International Journal of Mobile Learning and Organisation, 1(1), 81–102. https://doi.org/10.1504/IJMLO.2007.011190
   Google Scholar
• Wadley, G., Carter, M., & Gibbs, M. (2015). Voice in virtual worlds: The design, use, and influence of voice chat in online play. Human–Computer Interaction, 30(3–4), 336–365. https://doi.org/10.1080/07370024.2014.987346
   Web of Science ®Google Scholar
• Walter, S., Wendt, C., Böhnke, J., Crawcour, S., Tan, J.-W., Chan, A., Limbrecht, K., Gruss, S., & Traue, H. C. (2014). Similarities and differences of emotions in human–machine and human–human interactions: What kind of emotions are relevant for future companion systems? Ergonomics, 57(3), 374–386. https://doi.org/10.1080/00140139.2013.822566
   PubMed Web of Science ®Google Scholar
• Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., & Zhang, W. (2022). A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion, 83–84, 19–52. https://doi.org/10.1016/j.inffus.2022.03.009
   Web of Science ®Google Scholar
• Zepf, S., Hernandez, J., Schmitt, A., Minker, W., & Picard, R. W. (2021). Driver emotion recognition for intelligent vehicles. ACM Computing Surveys, 53(3), 1–30. https://doi.org/10.1145/3388790
   Web of Science ®Google Scholar
• Wu, W., Mitchell, P., & Lv, Y. (2023). Consistency in personality trait judgments across online chatting and offline conversation. Frontiers in Psychology, 14, 1077458. https://doi.org/10.3389/fpsyg.2023.1077458
   PubMed Web of Science ®Google Scholar
• Zhang, M., Wang, F., & Zhang, D. (2020). Individual differences in trait creativity moderate the state-level mood-creativity relationship. PLoS One, 15(8), e0236987. https://doi.org/10.1371/journal.pone.0236987
   PubMed Web of Science ®Google Scholar