Introduction to the Special Issue on Human–Computer Interaction Innovations in China

HCI in China started focusing on ergonomic aspects of computing technology. In the mid-1980s, engineering psychologists at Hangzhou University (now Zhejiang University), led by Prof. Zuxiang Zhu, initiated the empirical investigation of the ergonomics design for computer-based visual display terminals (VDT) and their impacts on human performance (Xu & Zhu, 1990; Zhu & Wu, 1990). The work was the first HCI-related research project funded by the National Science Foundation of China. During the early exploration, engineering psychologists at the Institute of Psychology under the China Academy of Science and Hangzhou University also addressed the unique HCI issues related to Chinese culture, namely, Chinese input to personal computers (Zhang & Chen, 1991; Zhu & Shen, 1991; Zhang et al., 1992). During the 1990s, HCI work was expanded to user interface design, multimodal technology, and related cognitive aspects, initiated by the interdisciplinary collaborative work between computer scientists at Peking University and the Institute of Software under the China Academy of Science as well as Hangzhou University (Dong et al., 1999).

As we entered the 2000s, many foreign corporations, such as Microsoft, IBM, Motorola, Intel, Siemens, and Nokia, established HCI research and development centers in China because of its huge talent pool and market potential, which had created significant influences on the HCI research and practice in China (Wang, 2003). Since then, HCI has been experiencing explosive growth in both Chinese industry and academia through interdisciplinary collaborations (Tian et al. 2019). The societal awareness of user experience for computing technology also injected new vitality into HCI practice in China. HCI is now becoming a thriving field in China. The HCI community is growing, including professionals from the fields of computer science, engineering psychology, human factors, user experience/usability, behavioral sciences, industrial design, etc.

Over the past decade, Chinese HCI professionals have made remarkable progress in HCI innovations. The country has emerged as a critical player in shaping the future of human-computer interaction and digital experiences. From cutting-edge research labs to vibrant startup ecosystems, China has fostered an environment conducive to HCI breakthroughs. Notable advances include the integration of modern technologies, such as intelligent technology, VR/AR/MR, and multimodal interaction technology, into user interfaces and digital solutions, resulting in more intuitive interactions and innovative solutions. More importantly, HCI research and application in China have significant impacts on human well-being and society. Chinese HCI professionals conduct research and application across the domains of healthcare, aerospace, aviation, information technology, transportation (autonomous driving), education, and so on. Thus, China’s HCI landscape is characterized by a fusion of innovative research and application with modern technology, traditional cultural elements, and interdisciplinary collaboration, creating societal impacts and culturally specific user experiences.

Given the transformative impact of China’s HCI progress, the Engineering Psychology Committee under the Chinese Psychological Society, supported by the Technical Committee of HCI under the China Computer Federation, has organized this special issue to explore how both international and Chinese communities benefit from these innovations. We have received a total of 88 submissions. Twenty-two of them have been accepted and included in this issue based on a rigorous peer review process. As a result, the contents of this issue comprise contributions from Chinese HCI professionals across academia and industry, showcasing original and high-quality HCI research and application across a variety of domains. By exploring these insights, we aim to provide a comprehensive understanding of the collaborative dynamics at play in the global HCI landscape, driven in part by China’s contributions.

The first article offers comprehensive reviews of HCI development in China over the last ten years. Shi, Du, et al. provide a leading overview of HCI's progress in China from the perspectives of both micro-HCI (i.e., focusing on the HCI design and technology) and macro-HCI (i.e., focusing on holistic aspects of HCI’s impacts on humans and society). By analyzing the achievements of HCI research and application (e.g., published papers and applications) by Chinese HCI professionals in the past ten years, this review showed that they not only worked on the understanding and modeling of humans, designing, and creating natural user interfaces but also focused on the psychological, behavioral and social impacts of new HCI paradigms and technologies. In addition, this review proposed that future research in the era of human-AI interaction should be grounded in understanding humans, which involves a shift in the HCI framework and associated methodologies to investigate the influence of human factors on human-computer interaction.

The next three articles focus on the emerging HCI issues when human interactions with intelligent systems in the AI era, such as trustworthy AI, affective computing, and human-robot interaction. Qi et al. developed an experimental paradigm (CAPTCHA recognition task) with high ecological validity to investigate how continuous errors in an automatic system and the timing of their occurrence affect human-automation trust. The results showed that trust is most affected when errors consecutively peak, revealing a significant peak-end effect on trust during human-computer interaction. This study provides new suggestions for a trustworthy AI design. As fundamental work for affective computing, Zhang et al. studied human emotional experience and evaluation by surveying a group of 400 participants to investigate HCI-specific emotional experiences and present a framework of emotional assessment for future HCI applications. Rau et al. explore the influence of a robot’s language and feedback framework on human-robot interaction, finding that different languages or feedback frameworks lead to varied performance and experiences. Notably, using a foreign language diminishes the impact of the robot’s feedback framework on participants’ performance and brain activation. The findings generate design implications for the development of intelligent robots and conversational AI agents.

HCI plays a crucial role in the development of AI-based autonomous vehicles. Currently, there are many HCI issues to be addressed. Wang et al. focus on enhancing driver perception through the Guided Warning of Invisible Danger from AR-HUD. Their experiments reveal improved situation awareness and reaction time in both driving modes, along with increased trust in auto-pilot. The study demonstrates that AR has the potential to help the driver become aware of invisible hazards through the AR-HUD, significantly enhancing driver performance. Tang et al. chose a novel perspective to investigate how situational games impact situation awareness and takeover performance. The results show that drivers engaged in situational games exhibit higher SA, better takeover performance, and less attention duration compared to drivers engaged in non-situational games.

HCI also plays a crucial role in healthcare, leading to better patient care and safety. Three articles contribute to HCI in healthcare. Non-pharmacological treatments have gained significant attention in the field of mild cognitive impairment (MCI). To close the gap in assessment methodology in existing rehabilitation training systems for MCI, Bu et al. propose a novel Leap Motion-based building block training system that incorporates system software capable of generating highly realistic virtual scenes, with the added capability of user behavior detection using Kinect. The results revealed significant differences between MCI cohort and the control cohort, offering an HCI approach for diagnosing and assessing MCI. Autism Spectrum Disorders (ASD) remain a healthcare challenge. To tackle this problem, Liu et al. employed both facial expressions and body poses as cues to recognize emotional states, while most traditional methods only leverage the former. The study evaluated this method on a challenging database collected by the team in real-world child–clinician interactive scenarios. The proposed method achieved significantly better results than baselines, offering a better HCI solution to assist in clinical practice. To aid in the rehabilitation of people with upper limb function impairments, Niu et al. explores the practicability of bimanual asymmetric interaction with Leap, a novel hand motion capture device. The study examines factors such as task difficulty, allocation, and interactive mode during the manipulation of virtual 3D objects. This research provides insights into how HCI can help design rehabilitative devices that interact with one healthy hand.

Space stations and aircraft cockpits are uniquely challenging HCI environments where astronauts and pilots interact with computerized space stations and cockpits. There are many special HCI issues significantly impacting safety. Wang et al. systematically examine the process where how HCI professionals played critical roles in contributing to the human factors design in the China Space Station (CSS). Specifically, the HCI professionals have carried out extensive studies on the human factors design and evaluation of the space manipulator teleoperation system, which has guided the optimization of space manipulator engineering. This article demonstrates how HCI work has contributed significantly to the safe and smooth completion of astronaut extravehicular activities on the CSS. In the context of aviation safety, Wang, Wang et al. conducted a quantitative assessment of human error risk, one of the critical HCI issues, in computerized aircraft cockpits. They analyzed performance shaping factors based on data from 872 aviation accident reports. The findings inform the development of a causality-based Human Reliability Analysis (HRA) method, offering guidance for preventing and mitigating human errors in modern computerized aircraft cockpits from a broader perspective beyond conventional HCI work on airborne pilot-cockpit interface design.

VR/AR/MR technologies offer immersive experiences and have potential applications in the context, such as the changes brought about by COVID-19 pandemic. HCI solutions should solve real problems for society. Feng, Wang et al. developed an MRLab that used wearable technologies such as a smart glove and head-mounted device to record sensory data and a multimodal hybrid fusion model GVVS to interpret the user’s experimental intent, which essentially transforms the user’s abstract behavioral actions into a probabilistic set of experimental intent that can be computed. The results show that the MRLab proved to be a useful alternative to traditional physics laboratory experiments during the pandemic. To advance the future VR/MR teaching environments, Zhang et al. investigated the creativity of future teachers in instructional design. They utilized VR/MR teaching environments to explore the mediating effect of flow, attention, and meditation on creativity levels. The empirical results indicate that more VR/MR experimental teaching environments would help cultivate the creativity level of future teachers in teaching design.

The next three articles explore the application of new HCI technologies to study traditional Chinese culture, which is the unique work done by Chinese HCI professionals. To leverage VR technology to support audience engaging user experience with cultural heritage (CH) proactively, such as the Dunhuang murals, Tong et al. explored an immersive VR reconstruction approach that can efficiently convert complex 2D mural art into 3D digital VR content while maintaining its originality. The 2D-to-VR reconstruction methods provided a VR reconstruction approach and insights for creating engaging user experiences and improving audience’s understanding of cultural and historical concepts while preserving the originality of the cultural heritage. Furthermore, Fu et al. developed an immersive virtual reality (IVR) system that provided users with learning, interactive, and entertaining experiences, facilitating their transition from mere visitors to active learners while engaging with digital CH. The Dunhuang Mural Restoration is used as a case study to demonstrate that the IVR system transforms passive visitors into engaged learners, significantly improving knowledge acquisition and conservation awareness. In addition, impressive progress has been made in developing companion socially interactive agents (SIAs). Current research on SIAs is mainly based on English-language databases. To close the gap, Xiang et al. developed the Chinese Natural Audiovisual Multimodal Database (CNAMD) corpus, the first and largest freely available Chinese multimodal database in multi-person interaction as the fundamental work for SIA research. The study tests the performance of six baselines on three tasks and proposes improvements for processing everyday Chinese data. The findings will help designers consider Chinese customs and characters more when designing a companion Chinese SIA, and make them more suitable for use in a Chinese cultural context.

The following four articles focus on enhancing user performance and experience using innovative VR/AR and interaction technology. Fang, Bao, et al. propose an innovative adaptive human-computer interface using the Adaptive Interface Design (AID) framework. It takes real-time user attention and system state as input, employs a hybrid entropy attention allocation model for decision-making, and adjusts real-time information saliency as output. The results show that the adaptive human-computer interface can effectively improve the supervisors’ performance and reduce the risk of system failure. Kuai et al. aimed to enhance the operational efficiency of head-gaze interaction in virtual reality (VR) by adjusting the control-display gain. Through four experiments, they discovered that implementing a dual-gain mode, with distinct control-display gains in the ballistic and corrective phases, effectively improved interaction efficiency without raising perceptual problems or increasing workload. Rui et al. explored the utilization of Extended Reality (XR) glasses for PC virtual extended displays. Their research, based on user interviews and usability tests, led to the development of a systematic user experience design and evaluation framework for XR virtual display technology. Finally, Xu et al. propose a visuo-tactile interaction design based on Norman’s design principles and spatial augmented reality (SAR) for cultural education. They applied this design to a Chinese chess app, SARChess, as a testbed to achieve integrated visual output and tangible input with seamless visuo-tactile consistency. The results show that SARChess has a significantly better user experience, learning performance, and player motivation. Design modeling tools and their experiences are essential for HCI design, significantly impacting the quality of products. Hou et al. proposed a two-stage user-based elicitation and multimodal interactive guiding grammar framework for virtual reality modeling scenarios. The experimental results demonstrate that this framework enhances interactions, making them more natural, efficient, easy to learn, and interesting compared with traditional 3D modeling. Feng, Zhou, et al. introduced FMASketch, a method for design sketching in a 3D AR space. This innovative method addresses depth perception challenges and resolves issues associated with observing content when drawing with a mobile device. The experiments verified the sketching method is easy to learn and provides a new idea for rapid modeling in AR scenarios.

In summary, these articles featured in this issue provide a glimpse into the evolving landscape of HCI in China over recent years. These contributions showcase not only innovative technological endeavors and promising outcomes but also highlight the challenges faced by China and the global community. This special issue aims to serve as a focal point for the scientific and industrial communities, encouraging attention to various HCI issues. By doing so, we aspire to enhance our collective ability to address the emerging challenges presented by intelligent technologies and human-AI collaboration in the intelligence era.

Acknowledgments

We would like to thank the IJHCI editors, Dr. Gavriel Salvendy and Dr. Constantine Stephanidis, for the opportunity to edit this country-based special issue and for their guidance along the way. We would also like to thank over 60 anonymous reviewers whose valuable insight helped to ensure the articles are rigorous to academia and relevant to industry practitioners.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Wei Xu

Wei Xu is a professor of Human Factors/HCI at Zhejiang University, China. He is a fellow of the International Ergonomics Association, the Human Factors and Ergonomics Society, and the Association for Psychological Science. His research interests include human-centered AI, human–AI interaction, and aviation human factors.

Feng Du

Feng Du is a professor at the Institute of Psychology, Chinese Academy of Sciences. He is the Chair of the Engineering Psychology Committee under the Chinese Psychological Society and a council member of the Chinese Ergonomics Society. His research interests include cognitive ergonomics, situation awareness, and human–AI trust.

Liang Zhang

Liang Zhang is an associate professor at the Institute of Psychology, Chinese Academy of Sciences. She is the deputy director of the Engineering Psychology Committee in Chinese Psychological Society. She applies a comprehensive methodology to assess variations in stress, emotional and cognitive states; and utilizes the findings to improve HCI.

Liezhong Ge

Liezhong Ge is a professor of HCI/Human Factors Engineering at the Center for Psychological Sciences of Zhejiang University, China. He is the former Chair of the Technical Committee of Engineering Psychology under the Chinese Psychological Society. His research interests include human–computer interaction, user experience, and facial recognition.