Human Computer Interaction on the Modern Flight Deck

ABSTRACT

There has been steady and significant growth and development of the technology available to pilots on the modern aviation flight deck. From improved information systems such as the Electronic Flight Bag (EFB), to increased control of the aircraft (e.g., automated collision avoidance maneuvering) and onwards to overall management of flights (via the Flight Management System, FMS), pilots now have a range of high-tech systems at their fingertips. This technology is designed to support pilots in maintaining situation awareness, making complex decisions, communicating effectively and assisting in flying the aircraft. However, each technological advancement comes with human computer interaction (HCI) considerations that warrant exploration. This special issue aims to bring together new knowledge and best practices, from the scientific community and industry, related to HCI issues on the modern flight deck. The goal of the special issue is to shed light on areas in need of improvement and potential new solutions for maintaining effective HCI on the flight deck. The articles within this special issue incorporate quantitative and qualitative research methods along with HCI design techniques to examine current and future use of technology on the flight deck and how it impacts pilot performance. Although this issue represents a small portion of the challenges pilots face when interacting with technology on the modern flight deck, it will hopefully serve as a lens to focus attention on the many HCI issues that pilots face, the progress that has been made to date, and the many areas that show potential for improvement.

There has been steady and significant growth and development of the technology available to pilots on the modern aviation flight deck. From improved information systems such as the Electronic Flight Bag (EFB) to increased control of the aircraft (e.g., automated collision avoidance maneuvering) and onwards to overall management of flights (via the Flight Management System, FMS), pilots now have a range of high-tech systems at their fingertips. This technology is designed to support pilots in maintaining situation awareness, making complex decisions, communicating effectively and assisting in flying the aircraft. However, each technological advancement in the cockpit comes with human–computer interaction (HCI) considerations that warrant exploration. While progress in regards to the use of technology on the flight deck has been an important part of a long-term trend toward increased safety in aviation (Boeing, 2020), it has also been part of setbacks in the form of incidents and accidents. In recent years this has been highlighted by the accidents of Lion Air Flight 610 (KNKT (Komite Nasional Keselamatan Trasportasi) – National Transportation Safety Committee of Indonesia, 2019) and Ethiopian Airlines Flight 302 (Accident Investigation Bureau (AAIB) of Ethiopia, 2020), in which HCI considerations have been central in the investigations. The same can be said of many other incidents and accidents, which demonstrates the importance of HCI issues on the flight deck for safe and efficient flight.

This special issue aims to bring together new knowledge and best practices, from the scientific community and industry, related to HCI issues on the modern flight deck. The goal of the special issue is to shed light on areas still in need of improvement and potential new solutions for maintaining effective HCI on the flight deck. This special issue contains eleven articles that incorporate quantitative and qualitative research methods along with HCI design techniques to examine current and future use of technology on the flight deck and how it impacts pilot performance.

The first four articles provide an examination of HCI issues related to the display of information to pilots. As more information has become available on the flight deck, the “how, what and where” of this information presentation must be considered to ensure effective and efficient use. In the first article, Lazaro, et al. examine how to effectively present information, conducting a series of three studies examining the visual complexity of cockpit displays and the effectiveness of various decluttering methods in order to improve visual search and target detection. Findings indicate that higher levels of visual complexity in flight deck displays, which can be attributed to the presence of multiple features, results in slower search times and more decision errors. Findings also indicated that the most effective declutter methods are dotting (i.e., reducing display elements to dots) and removal of display elements, compared to dimming and small sizing, which was the least effective. The authors provide guidance for system designers based on these findings. What information is removed in order to declutter can also have impacts on a pilot’s situation awareness, and in the second article, Yeh, et al., examine what pieces of information should be decluttered in a specific aviation display type: electronic charts. The authors collected survey data from a large sample of air transport, corporate, military, and general aviation pilots, and in conjunction with subject matter expert input, developed a classification scheme and prototype display concept for electronic chart information elements that need to be (a) displayed at all times, (b) displayed initially and toggled off/on, (c) not displayed initially and toggled on/off, or (d) not displayed at all. Presenting the right information is also critical for effective decision-making. In the third article, Parnell, et al. utilize the Perceptual Cycle Model to theoretically ground data collected from interviews with airline pilots to identify what information should be presented to pilots regarding engine status during an engine failure event, as well as pilot preferences for how information is presented, in general. Based on their findings, the authors provide recommendations for how future avionics systems could present such information. Another important consideration for new flight deck displays is where information is presented and how location impacts trust in, and use of, the information. In the fourth article, Carroll, et al. conducted a simulation study with Boeing 737 airline pilots to examine how a change in location of uncertified EFB information, from a personal electronic device, into the aircraft display panel, impacts pilot response to an information conflict between the EFB and approved sources of information on the flight deck. The results indicate that, as the EFB information became more integrated into the aircraft display panel, participants were more likely to both detect and investigate information conflicts, indicating that location matters and integration into the aircraft display panel may blur the line between certified and uncertified systems.

The next two articles present evaluations of emerging interface technology concepts in their ability to convey critical information to assist pilots in challenging situations such as loss of control (LOC) and upset recovery situations, which are a focus in the aviation industry. In the fifth article, Ellis, et al. present a study examining the effectiveness of a synthetic vision system, which provides a computer-generated image of the external scene topography as the background to the primary flight display, at improving flight crew awareness of airplane attitude and promoting effective upset recovery. The study compared pilot performance, workload, situation awareness and display reactions while responding to unusual attitude recovery events in a Boeing 787 full motion simulation. The results indicated that compared to traditional displays, the synthetic vision display led to significant benefits to the flight crew, including reductions in workload and increased situation awareness. In the sixth article, Van Baelen et al. present and evaluate three haptic feedback designs to support pilots in understanding where the aircraft is with respect to flight envelope protection limits. The evaluations resulted in preliminary findings that force feedback cues appear to be the most effective way to communicate this information to the pilots and can lead to increased safety margins. Technology such as this may prove important given recent fatal accidents involving pilot mode confusion associated with flight envelope protection, e.g., Air France Flight 447 (Bureau d’Enquêtes et d’Analyses pour la sécurité de l’aviation civile (BEA) – Civil Aviation Accident Investigation Bureau of France, 2012), Air Asia Flight 8501 (KNKT (Komite Nasional Keselamatan Trasportasi) – National Transportation Safety Committee of Indonesia, 2014), Lion Air Flight 610 (KNKT (Komite Nasional Keselamatan Trasportasi) – National Transportation Safety Committee of Indonesia, 2019) and Ethiopian Airlines Flight 302 (Accident Investigation Bureau (AAIB) of Ethiopia, 2020).

The next two articles present evaluations of flight deck interaction methods that utilize touch screen technology and some of the considerations necessary to facilitate effective interaction. As touch screen displays are explored, consideration must be taken regarding how to effectively design systems to support effective performance under turbulent conditions. In the seventh article, Wynne et al. examine the impact of various levels of turbulence on control inputs for various flight deck-relevant, touch screen interactions. Findings indicate that turbulence has differential impacts on different touch screen interaction techniques such as tapping and dragging, and that different interaction techniques are better suited for different types of tasks, such as number entry and waypoint panning. The article provides guidance for flight deck touchscreen system designers. In the eighth article, Khoshnewiszadeh and Pool present a promising approach for mitigating the effects of turbulence on touchscreen interactions through the use of an algorithm that provides a correction to touchscreen inputs, which cancels biodynamic feedthrough based on accelerations experienced by the aircraft. The authors provide empirical support for this method and a discussion of the feasibility of integrating it into current aircraft.

The last three articles deal explicitly with automation and some of the unique challenges that interaction with automation brings to the flight deck. Some of the key HCI problems that have plagued the aviation industry for decades are mode confusion and automation surprise. As aircraft have become more automated, this problem has continued to result in fatal aircraft accidents such as Air Inter 148, Flash Airlines Flight 604 Asiana 214 and many others. In the ninth article, Mumaw presents a unique alternative to redesigning automation within the certified systems in the panel: a supplemental system to aid in interpretation of “what the aircraft is doing”. This article illustrates how the design is grounded in the evolution of industry recommendations over recent decades and provides preliminary evaluation data regarding interface utility and usability. In the tenth article, Tokadli, Dorneich, and Matessa present a novel concept for an interface designed to facilitate human-machine teamwork, between the pilot and the automation. In this article, the authors present a display concept that is designed to allow a pilot, in single-pilot operations, to dynamically allocate functions to an automated system in real-time, to facilitate more effective human automation teamwork. The authors conducted a study that provides insight into pilot response to such an interface and interface attributes necessary for effective interaction. In the final article, Soo, Mavin and Kikkawa present an alternative to interface redesign to help with human-automation interaction, with the aim of improving pilot training. The authors present a longitudinal study that utilized airline pilot interviews and observations to examine the role that training plays in preparing pilots for working with flight deck automation. Through thematic analysis, the study identified eight themes underlying pilot learning processes related to automation and four key problems pilots face when learning to interact with automation. The authors provide recommended interventions for preparing pilots for working effectively with automation, including training that is aimed at transforming the way pilots think about automation, from being” just another aircraft system”, to being an automated team member in the cockpit.

The articles presented in this issue represent a small portion of the challenges pilots face when interacting with technology on the modern flight deck. This special issue will hopefully serve as a lens to focus attention, from both the scientific community and industry, on the many HCI issues that pilots face, including the progress that has been made to date and the many areas that show potential for improvement.

Acknowledgments

We would like to thank the IJHCI editors, Dr. Gavriel Salvendy and Dr. Constantine Stephanidis, for the opportunity to edit this special issue and their guidance along the way. We would also like to thank the many reviewers whose valuable insight helped to ensure both academic rigor of the articles and relevance to industry practitioners.

Additional information

Notes on contributors

Meredith Carroll

Meredith Carroll is an Associate Professor of Aviation Human Factors at Florida Institute of Technology’s College of Aeronautics. She has over 15 years of experience, both in industry and academia, studying human performance, human-computer interaction and learning in complex systems within commercial aviation, military and space applications.

Nicklas Dahlstrom

Nicklas Dahlstrom is Human Factors Manager at Emirates Airline in Dubai, United Arab Emirates, and Assistant Professor at Lund University School of Aviation, Sweden. He has more than three decades of experience in industry and academia, with research focused on pilot performance and training, especially on mental workload and simulation.