1,027
Views
0
CrossRef citations to date
0
Altmetric
Research Articles

From Video to Hybrid Simulator: Exploring Affective Responses toward Non-Verbal Pedestrian Crossing Actions Using Camera and Physiological Sensors

ORCID Icon, , , , &
Pages 3213-3236 | Received 26 Dec 2022, Accepted 06 Jun 2023, Published online: 04 Jul 2023

References

  • Arhin, S. A., Gatiba, A., Anderson, M., Manandhar, B., Ribbisso, M., & Acheampong, E. (2022). Effectiveness of modified pedestrian crossing signs in an urban area. Journal of Traffic and Transportation Engineering (English Edition), 9(1), 21–32. https://doi.org/10.1016/j.jtte.2021.04.001
  • Balconi, M., & Bortolotti, A. (2012). Empathy in cooperative versus non-cooperative situations: The contribution of self-report measures and autonomic responses. Applied Psychophysiology and Biofeedback, 37(3), 161–169. https://doi.org/10.1007/s10484-012-9188-z
  • Barrett, L. F. (2017). The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(11), 1833–1833. https://doi.org/10.1093/scan/nsx060
  • Bazilinskyy, P., Kooijman, L., Dodou, D., Mallant, K., Roosens, V., Middelweerd, M., Overbeek, L., & de Winter, J. (2022). Get out of the way! Examining eHMIs in critical driver-pedestrian encounters in a coupled simulator. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 360–371). https://doi.org/10.1145/3543174.3546849
  • Bethge, D., Kosch, T., Grosse-Puppendahl, T., Chuang, L. L., Kari, M., Jagaciak, A., & Schmidt, A. (2021). Vemotion: Using driving context for indirect emotion prediction in real-time. In The 34th Annual ACM Symposium on User Interface Software and Technology (pp. 638–651). Virtual Event USA; ACM. https://doi.org/10.1145/3472749.3474775
  • Bindschädel, J., Krems, I., & Kiesel, A. (2021). Interaction between pedestrians and automated vehicles: Exploring a motion-based approach for virtual reality experiments. Transportation Research Part F: Traffic Psychology and Behaviour, 82, 316–332. https://doi.org/10.1016/j.trf.2021.08.018
  • Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49–59. https://doi.org/10.1016/0005-7916(94)90063-9
  • Braun, M., Schubert, J., Pfleging, B., & Alt, F. (2019). Improving driver emotions with affective strategies. Multimodal Technologies and Interaction, 3(1), 21. https://doi.org/10.3390/mti3010021
  • Braun, M., Weber, F., & Alt, F. (2022). Affective automotive user interfaces? Reviewing the state of driver affect research and emotion regulation in the car. ACM Computing Surveys, 54(7), 1–26. https://doi.org/10.1145/3460938
  • Chou, K.-L., Lee, T., & Ho, A. H. (2007). Does mood state change risk taking tendency in older adults? Psychology and Aging, 22(2), 310–318. https://doi.org/10.1037/0882-7974.22.2.310
  • Clark, E. A., Kessinger, J., Duncan, S. E., Bell, M. A., Lahne, J., Gallagher, D. L., & O'Keefe, S. F. (2020). The facial action coding system for characterization of human affective response to consumer product-based stimuli: A systematic review. Frontiers in Psychology, 11, 920. https://doi.org/10.3389/fpsyg.2020.00920
  • Dawson, M. E., Schell, A. M., & Filion, D. L. (2016). The electrodermal system. In Handbook of psychophysiology (4th ed., pp. 217–243). Cambridge University Press. https://doi.org/10.1017/9781107415782.010
  • Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3–4), 169–200. https://doi.org/10.1080/02699939208411068
  • Ekman, P., & Friesen, W. V. (1978). Facial action coding system: Investigator’s guide. Consulting Psychologists Press.
  • Fleureau, J., Guillotel, P., & Orlac, I. (2013). Affective benchmarking of movies based on the physiological responses of a real audience. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (pp. 73–78). IEEE.
  • Fox, A. S., Lapate, R. C., Shackman, A. J., & Davidson, R. J. (2018). The nature of emotion: Fundamental questions. Oxford University Press.
  • Ghosh, S., Pons Rodriguez, G., Rao, S., El Ali, A., & Cesar, P. (2022). Exploring emotion responses toward pedestrian crossing actions for designing in-vehicle empathic interfaces. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1–6). Association for Computing Machinery. https://doi.org/10.1145/3491101.3519764
  • Golland, Y., Arzouan, Y., & Levit-Binnun, N. (2015). The mere co-presence: Synchronization of autonomic signals and emotional responses across co-present individuals not engaged in direct interaction. PLOS One, 10(5), e0125804. https://doi.org/10.1371/journal.pone.0125804
  • Gueguen, N., Eyssartier, C., & Meineri, S. (2016). A pedestrian’s smile and drivers’ behavior: When a smile increases careful driving. Journal of Safety Research, 56, 83–88. https://doi.org/10.1016/j.jsr.2015.12.005
  • Guéguen, N., Meineri, S., & Eyssartier, C. (2015). A pedestrian’s stare and drivers’ stopping behavior: A field experiment at the pedestrian crossing. Safety Science, 75, 87–89. https://doi.org/10.1016/j.ssci.2015.01.018
  • Habibovic, A., Lundgren, V. M., Andersson, J., Klingegård, M., Lagström, T., Sirkka, A., Fagerlönn, J., Edgren, C., Fredriksson, R., Krupenia, S., Saluäär, D., & Larsson, P. (2018). Communicating intent of automated vehicles to pedestrians. Frontiers in Psychology, 9, 1336. https://doi.org/10.3389/fpsyg.2018.01336
  • Haklay, M. M., & Weber, P. (2008). Openstreetmap: User-generated street maps. IEEE Pervasive Computing, 7(4), 12–18. https://doi.org/10.1109/MPRV.2008.80
  • Healey, J., & Picard, R. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156–166. https://doi.org/10.1109/TITS.2005.848368
  • Heilbron, F. C., & Niebles, J. C. (2014). Collecting and annotating human activities in web videos. In Proceedings of International Conference on Multimedia Retrieval (pp. 377–384). Association for Computing Machinery. https://doi.org/10.1145/2578726.2578775
  • Helm, J. L., Sbarra, D., & Ferrer, E. (2012). Assessing cross-partner associations in physiological responses via coupled oscillator models. Emotion, 12(4), 748–762. https://doi.org/10.1037/a0025036
  • Hoch, S., Althoff, F., McGlaun, G., & Rigoll, G. (2005). Bimodal fusion of emotional data in an automotive environment. In Proceedings.(ICASSP’05). IEEE international conference on acoustics, speech, and signal processing, 2005 (Vol. 2, pp. ii–1085). IEEE.
  • Hu, H., Zhu, Z., Gao, Z., & Zheng, R. (2018). Analysis on biosignal characteristics to evaluate road rage of younger drivers: A driving simulator study. In 2018 IEEE Intelligent Vehicles Symposium (IV) (pp. 156–161). https://doi.org/10.1109/IVS.2018.8500444
  • Ihme, K., Dömeland, C., Freese, M., & Jipp, M. (2018). Frustration in the face of the driver: A simulator study on facial muscle activity during frustrated driving. Interaction Studies, 19(3), 487–498. https://doi.org/10.1075/is.17005.ihm
  • Ihme, K., Unni, A., Zhang, M., Rieger, J. W., & Jipp, M. (2018). Recognizing frustration of drivers from face video recordings and brain activation measurements with functional near-infrared spectroscopy. Frontiers in Human Neuroscience, 12, 327. https://doi.org/10.3389/fnhum.2018.00327
  • Jeon, M. (2015). Towards affect-integrated driving behaviour research. Theoretical Issues in Ergonomics Science, 16(6), 553–585. https://doi.org/10.1080/1463922X.2015.1067934
  • Jeon, M. (2016). Don’t cry while you’re driving: Sad driving is as bad as angry driving. International Journal of Human–Computer Interaction, 32(10), 777–790. https://doi.org/10.1080/10447318.2016.1198524
  • Jeon, M., Yim, J.-B., & Walker, B. N. (2011). An angry driver is not the same as a fearful driver: Effects of specific negative emotions on risk perception, driving performance, and workload. In Proceedings of the 3rd International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 137–142). ACM.
  • Koch, K., Mishra, V., Liu, S., Berger, T., Fleisch, E., Kotz, D., & Wortmann, F. (2021). When do drivers interact with in-vehicle well-being interventions? An exploratory analysis of a longitudinal study on public roads. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(1), 1–30. https://doi.org/10.1145/3448116
  • Kooij, J. F. P., Schneider, N., Flohr, F., & Gavrila, D. M. (2014). Context-based pedestrian path prediction. In European Conference on Computer Vision (pp. 618–633). LNCS.
  • Kotseruba, I., Rasouli, A., & Tsotsos, J. K. (2016). Joint attention in autonomous driving (JAAD). arXiv preprint arXiv160904741, 264–269.
  • Likas, A., Vlassis, N., & Verbeek, J. J. (2003). The global k-means clustering algorithm. Pattern Recognition, 36(2), 451–461. https://doi.org/10.1016/S0031-3203(02)00060-2
  • Lutz, A., Brefczynski-Lewis, J., Johnstone, T., & Davidson, R. J. (2008). Regulation of the neural circuitry of emotion by compassion meditation: Effects of meditative expertise. PLOS One, 3(3), e1897. https://doi.org/10.1371/journal.pone.0001897
  • Ma, Z., Mahmoud, M., Robinson, P., Dias, E., & Skrypchuk, L. (2017). Automatic detection of a driver’s complex mental states. In International Conference on Computational Science and Its Applications (pp. 678–691). Springer.
  • Magaña, V. C., Scherz, W. D., Seepold, R., Madrid, N. M., Pañeda, X. G., & Garcia, R. (2020). The effects of the driver’s mental state and passenger compartment conditions on driving performance and driving stress. Sensors, 20(18), 5274. https://doi.org/10.3390/s20185274
  • Mahadevan, K., Somanath, S., & Sharlin, E. (2018). Communicating awareness and intent in autonomous vehicle-pedestrian interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 429). Associated Computing Machinery.
  • Malta, L., Angkititrakul, P., Miyajima, C., & Takeda, K. (2008). Multi-modal real-world driving data collection, transcription, and integration using bayesian network. In 2008 IEEE intelligent vehicles symposium (pp. 150–155). ACM.
  • Masip, D., North, M. S., Todorov, A., & Osherson, D. N. (2014). Automated prediction of preferences using facial expressions. PLOS One, 9(2), e87434. https://doi.org/10.1371/journal.pone.0087434
  • Megías, A., Maldonado, A., Cándido, A., & Catena, A. (2011). Emotional modulation of urgent and evaluative behaviors in risky driving scenarios. Accident; Analysis and Prevention, 43(3), 813–817. https://doi.org/10.1016/j.aap.2010.10.029
  • Mehrabian, A. (1996). Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology, 14(4), 261–292. https://doi.org/10.1007/BF02686918
  • Mesken, J., Hagenzieker, M. P., Rothengatter, T., & de Waard, D. (2007). Frequency, determinants, and consequences of different drivers’ emotions: An on-the-road study using self-reports,(observed) behaviour, and physiology. Transportation Research Part F: Traffic Psychology and Behaviour, 10(6), 458–475. https://doi.org/10.1016/j.trf.2007.05.001
  • Munla, N., Khalil, M., Shahin, A., & Mourad, A. (2015). Driver stress level detection using HRV analysis. In 2015 International Conference on Advances in Biomedical Engineering (ICABME) (pp. 61–64). IEEE. https://doi.org/10.1109/ICABME.2015.7323251
  • Nason, G. P., & Silverman, B. W. (1995). The stationary wavelet transform and some statistical applications. In Wavelets and statistics (pp. 281–299). Springer.
  • Nathanael, D., Portouli, E., Papakostopoulos, V., Gkikas, K., Amditis, A. (2018). Naturalistic observation of interactions between car drivers and pedestrians in high density urban settings. In Congress of the International Ergonomics Association (pp. 389–397).
  • Paschero, M., Del Vescovo, G., Benucci, L., Rizzi, A., Santello, M., Fabbri, G., Mascioli, F. F. (2012). A real time classifier for emotion and stress recognition in a vehicle driver. In 2012 IEEE International Symposium on Industrial Electronics (pp. 1690–1695). IEEE.
  • Pfleging, B., Fekety, D. K., Schmidt, A., & Kun, A. L. (2016). A model relating pupil diameter to mental workload and lighting conditions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5776–5788). ACM. https://doi.org/10.1145/2858036.2858117
  • Picard, R. W. (2000). Affective computing. MIT Press.
  • Plutchik, R. (2001). The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist, 89(4), 344–350. https://doi.org/10.1511/2001.28.344
  • Ranasinghe, C., Holländer, K., Currano, R., Sirkin, D., Moore, D., Schneegass, S., & Ju, W. (2020). Autonomous vehicle-pedestrian interaction across cultures: Towards designing better external human machine interfaces (eHMIs). In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–8). https://doi.org/10.1145/3334480.3382957
  • Ranganathan, P., Pramesh, C., & Aggarwal, R. (2017). Common pitfalls in statistical analysis: Measures of agreement. Perspectives in Clinical Research, 8(4), 187–191. https://doi.org/10.4103/picr.PICR_123_17
  • Rao, S., Ghosh, S., Pons Rodriguez, G., Röggla, T., El Ali, A., & Cesar, P. (2022). Investigating affective responses toward in-video pedestrian crossing actions using camera and physiological sensors. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 226–235). https://doi.org/10.1145/3543174.3546842
  • Rasouli, A., Kotseruba, I., & Tsotsos, J. K. (2017a). Agreeing to cross: How drivers and pedestrians communicate. In 2017 IEEE Intelligent Vehicles Symposium (IV) (pp. 264–269). IEEE. https://doi.org/10.1109/IVS.2017.7995730
  • Rasouli, A., Kotseruba, I., & Tsotsos, J. K. (2017b). Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior. In Proceedings of the IEEE International Conference on Computer Vision (pp. 206–213). IEEE.
  • Ren, Z., Jiang, X., & Wang, W. (2016). Analysis of the influence of pedestrians’ eye contact on drivers’ comfort boundary during the crossing conflict. Procedia Engineering, 137, 399–406. https://doi.org/10.1016/j.proeng.2016.01.274
  • Rigas, G., Goletsis, Y., & Fotiadis, D. I. (2012). Real-time driver’s stress event detection. IEEE Transactions on Intelligent Transportation Systems, 13(1), 221–234. https://doi.org/10.1109/TITS.2011.2168215
  • Risto, M., Emmenegger, C., Vinkhuyzen, E., Cefkin, M., & Hollan, J. (2017). Human-vehicle interfaces: The power of vehicle movement gestures in human road user coordination. In Proceedings of the Ninth International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design (pp. 411–420). National Advanced Driving Simulator. https://doi.org/10.17077/drivingassessment.1633
  • Roidl, E., Frehse, B., & Höger, R. (2014). Emotional states of drivers and the impact on speed, acceleration and traffic violations—A simulator study. Accident; Analysis and Prevention, 70, 282–292. https://doi.org/10.1016/j.aap.2014.04.010
  • Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/h0077714
  • Sani, S. R. H., Tabibi, Z., Fadardi, J. S., & Stavrinos, D. (2017). Aggression, emotional self-regulation, attentional bias, and cognitive inhibition predict risky driving behavior. Accident; Analysis and Prevention, 109(190), 78–88. https://doi.org/10.1016/j.aap.2017.10.006
  • Sarkar, P., & Etemad, A. (2020). Self-supervised learning for ECG-based emotion recognition. In ICASSP 2020 – 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3217–3221). IEEE. https://doi.org/10.1109/ICASSP40776.2020.9053985
  • Schmidt, S., & Färber, B. (2009). Pedestrians at the kerb–recognising the action intentions of humans. Transportation Research Part F: Traffic Psychology and Behaviour, 12(4), 300–310. https://doi.org/10.1016/j.trf.2009.02.003
  • Schmidt-Daffy, M. (2012). Velocity versus safety: Impact of goal conflict and task difficulty on drivers’ behaviour, feelings of anxiety, and electrodermal responses. Transportation Research Part F: Traffic Psychology and Behaviour, 15(3), 319–332. https://doi.org/10.1016/j.trf.2012.02.004
  • Schneemann, F., & Gohl, I. (2016). Analyzing driver-pedestrian interaction at crosswalks: A contribution to autonomous driving in urban environments. In 2016 IEEE Intelligent Vehicles Symposium (IV) (pp. 38–43). https://doi.org/10.1109/IVS.2016.7535361
  • Schuller, B., Wimmer, M., Arsic, D., Moosmayr, T., & Rigoll, G. (2008). Detection of security related affect and behaviour in passenger transport. In Ninth annual conference of the international speech communication association (p. 265–268). IEEE.
  • Schulz, A. T., & Stiefelhagen, R. (2015). Pedestrian intention recognition using latent-dynamic conditional random fields. In 2015 IEEE Intelligent Vehicles Symposium (IV) (pp. 622–627). IEEE. https://doi.org/10.1109/IVS.2015.7225754
  • Shi, Y., Ruiz, N., Taib, R., Choi, E., & Chen, F. (2007). Galvanic skin response (GSR) as an index of cognitive load. In CHI ’07 Extended Abstracts on Human Factors in Computing Systems (pp. 2651–2656). Association for Computing Machinery. https://doi.org/10.1145/1240866.1241057
  • Singh, R. R., Conjeti, S., & Banerjee, R. (2013). A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals. Biomedical Signal Processing and Control, 8(6), 740–754. https://doi.org/10.1016/j.bspc.2013.06.014
  • Sucha, M., Dostal, D., & Risser, R. (2017). Pedestrian-driver communication and decision strategies at marked crossings. Accident; Analysis and Prevention, 102(107050), 41–50. https://doi.org/10.1016/j.aap.2017.02.018
  • Taib, R., Tederry, J., & Itzstein, B. (2014). Quantifying driver frustration to improve road safety. In CHI’14 Extended Abstracts on Human Factors in Computing Systems (pp. 1777–1782). ACM. https://doi.org/10.1145/2559206.2581258
  • Vanutelli, M. E., Gatti, L., Angioletti, L., & Balconi, M. (2017). Affective synchrony and autonomic coupling during cooperation: A hyperscanning study. BioMed Research International, 2017(14), 1–9. https://doi.org/10.1155/2017/3104564
  • Wang, C., & Cesar, P. (2017). The play is a hit: But how can you tell? In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (pp. 336–347). Association for Computing Machinery. https://doi.org/10.1145/3059454.3059465
  • Wang, C.-A., Baird, T., Huang, J., Coutinho, J. D., Brien, D. C., & Munoz, D. P. (2018). Arousal effects on pupil size, heart rate, and skin conductance in an emotional face task. Frontiers in Neurology, 9, 1029. https://doi.org/10.3389/fneur.2018.01029
  • Wang, T., Wu, J., Zheng, P., & McDonald, M. (2010). Study of pedestrians’ gap acceptance behavior when they jaywalk outside crossing facilities. In 13th International IEEE Conference on Intelligent Transportation Systems (pp. 1295–1300). IEEE.
  • Wang, Y., Hespanhol, L., Worrall, S., & Tomitsch, M. (2022). Pedestrian-vehicle interaction in shared space: Insights for autonomous vehicles. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 330–339). https://doi.org/10.1145/3543174.3546838
  • Weber, F., Chadowitz, R., Schmidt, K., Messerschmidt, J., Fuest, T. (2019). Crossing the street across the globe: A study on the effects of eHMI on pedestrians in the US, Germany and China. In International Conference on Human–Computer Interaction (pp. 515–530).
  • Wu, B.-F., Chu, Y.-W., Huang, P.-W., Chung, M.-L., & Lin, T.-M. (2017). A motion robust remote-PPG approach to driver’s health state monitoring. In C.-S. Chen, J. Lu, & K.-K. Ma (Eds.), Computer Vision – ACCV 2016 Workshops (pp. 463–476). Springer International Publishing.
  • Yang, S. (2017). Driver behavior impact on pedestrians’ crossing experience in the conditionally autonomous driving context.
  • Zadeh, A., Chong Lim, Y., Baltrusaitis, T., & Morency, L.-P. (2017). Convolutional experts constrained local model for 3D facial landmark detection. In Proceedings of the IEEE International Conference on Computer Vision Workshops (pp. 2519–2528).
  • Zepf, S., Dittrich, M., Hernandez, J., & Schmitt, A. (2019). Towards empathetic car interfaces: Emotional triggers while driving. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–6). ACM.
  • Zepf, S., Hernandez, J., Schmitt, A., Minker, W., & Picard, R. W. (2021). Driver emotion recognition for intelligent vehicles: A. ACM Computing Surveys, 53(3), 1–30. https://doi.org/10.1145/3388790
  • Zhao, Y., Miyahara, T., Mizuno, K., Ito, D., & Han, Y. (2021). Analysis of car driver responses to avoid car-to-cyclist perpendicular collisions based on drive recorder data and driving simulator experiments. Accident; Analysis and Prevention, 150, 105862. https://doi.org/10.1016/j.aap.2020.105862
  • Zhi, R., Liu, M., & Zhang, D. (2020). A comprehensive survey on automatic facial action unit analysis. The Visual Computer, 36(5), 1067–1093. https://doi.org/10.1007/s00371-019-01707-5
  • Zimasa, T., Jamson, S., & Henson, B. (2019). The influence of driver’s mood on car following and glance behaviour: Using cognitive load as an intervention. Transportation Research Part F: Traffic Psychology and Behaviour, 66, 87–100. https://doi.org/10.1016/j.trf.2019.08.019