Ironies of artificial intelligence

Abstract

Bainbridge’s Ironies of Automation was a prescient description of automation related challenges for human performance that have characterised much of the 40 years since its publication. Today a new wave of automation based on artificial intelligence (AI) is being introduced across a wide variety of domains and applications. Not only are Bainbridge’s original warnings still pertinent for AI, but AI’s very nature and focus on cognitive tasks has introduced many new challenges for people who interact with it. Five ironies of AI are presented including difficulties with understanding AI and forming adaptations, opaqueness in AI limitations and biases that can drive human decision biases, and difficulties in understanding the AI reliability, despite the fact that AI remains insufficiently intelligent for many of its intended applications. Future directions are provided to create more human-centered AI applications that can address these challenges.

Practitioner summary:

Artificial Intelligence (AI) creates many new challenges for human interaction. Five ironies of AI are discussed that limit its ultimate success, and future directions are provided to create more human-centered AI applications that can address these challenges.

Keywords:

• Automation
• artificial intelligence
• human-centered AI
• situation awareness
• bias
• transparency

Introduction

Lisanne Bainbridge’s 1983 paper, the Ironies of Automation (Bainbridge 1983), was a telling and prescient summary of the many challenges that arise from automation. She pointed out the ways in which automation, paradoxically, make the human’s job more crucial and more difficult, rather than easier and less essential as so many engineers believe. Not only does automation introduce new design errors into the control of systems, but it creates very different jobs that have many new problems, with the result that people may be less able to perform when needed. They need to be more skilled to understand and operate the automation, while simultaneously the automation leads to skill atrophy. Additional system complexity is introduced as well as vigilance problems that interfere with peoples’ ability to oversee the automation. And while manual workload may be decreased much of the time, cognitive workload is often increased at critical times.

After 40 years, Bainbridge’s keen observations continue to hold true as the use of automation has increased across many domains, including aviation, air traffic control, automated process control, drilling, and transportation systems. Problems with loss of skills needed for manual performance and decision-making have been reported in aviation (Jacobson 2010; National Transportation Safety Board 2010; Wiener and Curry 1980), information automation (Volz et al. 2016), and vehicle automation (Nordhoff et al. 2023), among others. Inadequate training on automation has been found to be a critical problem associated with many aviation automation accidents (Funk et al. 1999), and attention to automation training has often been lacking (Strauch 2017). Increased cognitive workload, particularly at critical times, has been shown to be problematic, making it quite challenging for pilots take over manual control when needed, leading to numerous aviation accidents involving automation (Endsley and Strauch 1997; Federal Aviation Administration 2019; Funk et al. 1999).

While automation can be very useful when it works well, of critical concern, when automation is not perfect, people struggle to compensate for its shortcomings. A wide body of evidence has amassed showing that automation decreases peoples’ ability to intervene when needed because it puts them out-of-the-loop (OOTL). One study found 26 automation-related accidents among major air carriers between 1972 and 2013 in which pilots were significantly challenged in understanding what the automation was doing and interacting with it correctly to avoid the resulting accident (Gawron 2019). Another review found that two of the biggest factors underlying automation accidents and incidents are inadequate understanding of automation and poor transparency of the behaviour of the automation (Funk et al. 1999). These problems are not limited to aviation; similar automation related OOTL failures have contributed to major accidents in power grid operations (U.S. Canada Power System Outage Task Force 2004) and oil drilling (National Academy of Engineering and National Research Council 2012), for example.

OOTL related accidents have been shown to be due to the loss of situation awareness (SA) that occurs when people monitor automated systems, which arise from a combination of poor information displays, vigilance and complacency problems, and loss of cognitive engagement (Endsley and Kiris 1995). Summarising a wide body of research examining this phenomenon over the past 30 years, I posited an automation conundrum which states that:

The more automation is added to a system, and the more reliable and robust that automation is, the less likely that human operators overseeing the automation will be aware of critical information and able to take over manual control when needed. More automation refers to automation use for more functions, longer durations, higher levels of automation, and automation that encompasses longer task sequences. (Endsley 2017b)

In related work, Chris Wickens described the lumberjack effect which states that while automation can create better performance in routine situations, it also can lead to more catastrophic outcomes in non-routine situations where automation is often ill suited (Sebok and Wickens 2017). Overall, working with automated systems has proven challenging for people. Not only has automation often been limited in terms of its reliability and the range of situations it can handle, but simultaneously it negatively effects the degree to which people have been able to compensate for its deficiencies when needed.

New ironies with artificial intelligence

In the newest wave of automation, the use of artificial intelligence (AI) software is being touted as the means for overcoming the past limitations of traditional software automation. AI is a form of automation that is directed at highly perceptual and cognitive tasks (National Academies of Sciences Engineering and Medicine 2021; U. S. Air Force 2015). The goal of AI is to perform tasks that would normally require human intelligence, such as recognising speech or objects in visual scenes, making decisions and solving problems. AI applications potentially have the ‘ability to reason, discover meaning, generalize, or learn from past experience’ (Copeland 2020), although current instantiations may only do small portions of this.

Earlier versions of AI software relied on knowledge engines with extensive rules for behaviour or cases that had to be derived and encoded manually. Today’s AI instead uses machine learning (ML) algorithms (such as neural networks) to detect and recognise patterns in data, largely benefiting from the availability of big data sets that are digitised to produce useful products. For example, digital twins rely heavily on ML to process large sets of data for understanding complex systems, performing fault diagnosis, and predicting system failures in manufacturing and aerospace applications (Rathore et al. 2021). In other cases, such as autonomous vehicles, extensive sets of sensors need to be employed to gather environmental information for processing by the AI. And in some cases, generative AI techniques are actually creating new content (such as articles or pictures) by recombining or iterating based on input examples.

As an advanced form of automation, many of the same challenges and ironies identified by Bainbridge equally apply to the job of working with an AI system. Deskilling, increases in system complexity, increases in cognitive workload, and decreases in SA remain fundamental problems to be addressed with AI systems. As long as people need to interact with and potentially correct for deficiencies of the AI system, these fundamental human interaction challenges remain. Changes in the mechanism for creating the automated behaviours do not in any way alleviate previous automation related concerns that result from the fundamental changes in the roles of people along with co-occurring changes to workload, SA and engagement.

While AI can be considered an advanced (and potentially more capable) form of automation, its reliance on learning algorithms also creates additional challenges for the people who interact with these systems. Most notably, AI based on learning techniques suffers from opaqueness; it can be quite difficult for people to understand how the AI works and what features of the environment or situation it is basing its performance on (or is oblivious to). (Endsley 2017b; Huang and Endsley 1997). In addition, the learning algorithms themselves create a new kind of challenge for people—that of maintaining a mental model of a constantly changing and evolving system (U. S. Air Force 2015). Thus, in addition to Bainbridge’s initial ironies of automation, several additional ironies are created with AI. I will first discuss these ironies, followed by a discussion of inter-related approaches for mitigating them.

Irony 1: Artificial intelligence is still not that intelligent

AI has come a long way in recent years. New generative AI systems, such as ChatGPT, can create quite reasonable text based on a natural language generator and behind-the-scenes data sets that may be useful in many tasks. Initial results indicate that people may have trouble determining whether an essay was generated by a human or the AI, indicating potentially Turing level intelligence. A deep examination, however, shows that even this impressive feat is limited. Its answers to questions may be inaccurate or biased and it lacks an understanding of context (Rudolph, Tan, and Tan 2023). It does not possess any ‘common sense, reasoning, intuition, or creativity’ according to the ChatGPT website. While possessing an excellent natural language capability, it still is only capable of rephrasing the information in its underlying database and predicting what words are most likely to come next. And it often may simply make up information if it does not know something (i.e. hallucinations)(Alkaissi and McFarlane 2023; Sallam 2023).

While ChatGPT is growing and improving rapidly, one may ask is this intelligence? Does intelligence require more than gathering and regurgitating information (even if that information is presented well)? An intelligent system is defined as one that recognise situations, adapts to changes and generates solutions to even novel problems, and can act to optimise performance (Copeland 2020). It can be argued that generative AI still does not provide these capabilities; it just provides an excellent mimic.

Although programs such as AlphaGo can actually out-perform humans via ML (Silver et al. 2017), this is a highly constrained problem without the wide variety of edge cases and novel situations that can occur in real world settings. ML is excellent for data analysis when there is a large data set of situations to learn from, however, it is far less capable in managing novel situations that pose the edge cases where human intelligence is better able to perform.

AI systems are also being proposed for far more difficult tasks with which it still struggles. Even though AI can operate fairly easily on the basis of already digitised data, AI that requires sensors and machine vision for information inputs still has problems with accuracy (Akhtar and Mian 2018; Feng et al. 2019) and is easily confused by noisy data. This creates significant difficulties for any system that must operate in the real (as opposed to virtual) world, such as flying, driving, military operations and power systems.

Most importantly, Pearl and Mackenzie (2018) point out that ML based AI systems lack a model of causation that is critical for predicting future events, simulating potential actions, or appropriately generalising to new situations. Decision making requires not only understanding what is currently happening, but also projecting what is likely to happen, or what could possibly happen; therefore, this is a serious limitation on AI performance. Ultimately, truly intelligent AI will require capabilities which exceed that provided by current pattern-recognition based ML approaches. Intelligence will require the ability to perform well in untrained situations, the ability to recognise the limits of its own abilities and perform gracefully in such cases, the ability to manage uncertainty including noisy data, and the ability to predict future events and apply general knowledge to new situations appropriately.

Irony 2: the more intelligent and adaptive the AI, the less able people are to understand the system

Mental models constitute an important mechanism that people use for understanding the functioning of any system, including automation and AI (Manktelow and Jones 1987; Rouse and Morris 1985; Wilson and Rutherford 1989). Mental models are used for gathering and interpreting information to form SA by directing attention to relevant information and by providing the mechanisms needed for integrating information to create comprehension and projection of future events (Endsley 1995).

Challenges in developing accurate mental models of automated systems are well documented (Mogford 1997; Mumaw, Sarter, and Wickens 2001; Phillips et al. 2011; Silva and Hansman 2015). Automation tends to be inherently complex and opaque, with deep logic structures, making it quite difficult for people to develop good mental models and SA (Endsley 1995). This leads to unexpected ‘automation surprise’ occurring in infrequently encountered conditions (McClumpha and James 1994; Vakil and Hansman 1997; Woods and Sarter 2000).

It is even more difficult to develop an accurate mental model of an AI system than of traditional automation. First, because modern AI systems are generally developed through ML, distinct logic or rules are not apparent. The way in which the system will perform in different circumstances is inherently opaque and not open to inspection. Even the developers of AI systems are often unaware of what features of the situation it is using and how, and of its limitations and capabilities for different circumstances (Ferreira and Monteiro 2020). Therefore, mental models of how AI works must be developed through experience, with users creating their own representations of AI capabilities and limitations to guide their expectations of what it will do in difference situations, often incorrectly it turns out (Druce et al. 2021).

In addition, maintaining an accurate mental model of AI is made much more difficult by the fact that it can change over time (Endsley 2017b; U. S. Air Force 2015). One of the advantages of using ML approaches to AI is that it can constantly improve and extend as it encounters new situations. This, however, only makes the challenge of maintaining an accurate mental model of the AI more difficult. If the logic and capabilities of the AI change over time, users may not even be aware of these changes or able to update their internal models appropriately, and training for AI systems is often lacking (Casner and Hutchins 2019; Endsley 2017a). Without accurate and up-to-date mental models of how AI works, the ability of people to predict and understand AI actions is severely limited, decreasing their ability to adopt effective strategies for dealing with any AI deficiencies.

Irony 3: the more capable the AI, the poorer people’s self-adaptive behaviours for compensating for shortcomings

People can often compensate for systems with limited reliability by paying more attention to operations at times when the system may have difficulties (Bagheri and Jamieson 2004). When people believe their SA is poor, they tend to behave adaptively, acting conservatively to avoid negative outcomes while gathering more information to improve their SA (Endsley 1995). A key challenge arises in that not only can automation and AI directly reduce SA, but it also compromises the very mechanisms that people rely on when they need to improve their SA.

People actually have significant problems in assessing the accuracy of their own SA. A recent meta-analysis of 37 studies found very poor correlation between objective measures of SA and subjective measures of SA (Endsley 2020). There are a number of reasons for this disconnect, including limited feedback, Kruger-Dunning effects (the tendency of some people to over-estimate their abilities), the use of different brain structures for confidence assessments, selective information sampling, lack of knowledge of missing information, poor mental models, and the probabilistic link between SA and outcomes that limits learning (Endsley 2020). People often simply do not know what they do not know.

With respect to AI, the degree to which people may not know when their SA is low limits their ability to direct more attention to overseeing the AI or taking over in situations that it is ill-suited for. A recent study of Tesla drivers found that 46% believed that the AI autopilot improved their SA, while simultaneously 45% reported increased complacency, 21% reported over-reliance on the autopilot, and 15% reported mind-wandering and fatigued driving (Nordhoff et al. 2023). Even though many drivers seemed aware of some of the negative effects of the autopilot that could degrade their SA, this was at direct odds with their belief that SA was improved.

This disconnect can be partially explained by the observation that SA may actually be better at times with an autopilot, due to an increased ability to look around, even though it may be worse at other times due to complacency, poor engagement and limited information on autopilot functioning (Endsley 2017a). That is, SA becomes more variable. However, because people are also poor at determining when their SA is low, they are unable to effectively alter their information gathering behaviours when needed (e.g. introduce more scanning or interrupt competing tasking) in order to compensate for these deficiencies. Complacency, mind-wandering, and an increase in engaging in competing tasks become commonplace problems (Carsten et al. 2012; Hergeth et al. 2016; Ma and Kaber 2005; Sethumadhavan 2009).

Even as AI decreases SA, simultaneously behaviours typically used to increase SA also become compromised. While this problem exists with all automation, it can be argued that it is worse with AI. First, because AI is a more capable and higher level of automation, able to operate in a wider set of circumstances than traditional automation, SA can become even more degraded per the automation conundrum (Endsley 2017b). Therefore, it is likely to happen more frequently with AI systems. Secondly, the inherent opaqueness of AI systems will exacerbate the inability of people to accurately calibrate their understanding of system operations.

Irony 4: the more intelligent the AI, the more obscure it is, and the less able people are to determine its limitations and biases and when to use the AI

The challenge of AI bias has received considerable attention. These biases often are introduced inadvertently through the provision of limited or statistically biased training sets (i.e. limited representativeness of problems) that create biases towards certain sets of conclusions (Daugherty and Wilson 2018; Gianfrancesco et al. 2018; West, Whittaker, and Crawford 2019), as well as artefacts that creep into the development of the AI algorithms (Northcutt, Athalye, and Mueller 2021). These biases create a shortcoming in the performance of AI in that it will perform more poorly or inaccurately in situations that are different than what it has been trained on. Problems with AI systems introducing racial or gender bias in hiring decisions or facial recognition have been widely publicised, for example (Howard and Borenstein 2018; Osoba & Welser IV, 2017). As a more general case, bias can be considered any use of an AI system in situations outside of its training (National Academies of Sciences Engineering and Medicine 2021), i.e. an over-generalization that occurs when AI trained to operate in certain conditions is applied in other conditions.

People are often expected to be able to compensate for AI shortcomings, like bias, substituting their own knowledge and judgement in cases where the AI may be deficient. Paradoxically, however, AI makes it very difficult to do so. First, these biases tend to be hidden due the opaque nature of ML. Even the developers of AI systems may not know what biases have inadvertently been introduced in the training process. Furthermore, the users of AI systems are generally a different set of people than the developers of the AI, and therefore are even less likely to understand the limitations of its training or what situations it should be limited to.

For example, AI based recommendation systems have been found to promote misinformation and conspiracy theories (Hussein, Juneja, and Mitra 2020; Mortimer 2017). As people search for information or view certain web sites on social media, the AI recommender systems encourage additional related content. People may mistakenly view the recommended ‘similar’ content as having undeserved authenticity. They are not aware of how the AI system works, what it bases its recommendations on, or the risks inherent in its output. These biases are hidden.

Secondly, humans do not form their decisions independently from AI, but are directly influenced by the provision of recommendations or assessments from the AI (Endsley and Jones 2012). People tend to anchor on the recommendation of the AI system, and then gather information to agree or disagree with it, creating confirmation bias. AI biases therefore can directly compound human biases in the decision process, thereby reducing the reliability of the joint human-AI system. Further, the impact of the AI biases can vary depending on the format and framing of the AI system’s recommendations (Banbury et al. 1998; Endsley and Kris 1994; Friesen et al. 2021; Selcon 1990). Rather than overcoming human decision bias, AI can make it worse through this process of anchoring and confirmation bias.

In that AI biases are generally invisible, unknown by both developers and users of systems, and they can affect human decision making quite surreptitiously, their negative effects can be insidious. Therefore, people will be often unable to detect and compensate for these biases (by choosing when to use the system or interjecting corrections, for example).

Irony 5: the more natural the AI communications, the less able people are to understand the trustworthiness of the AI

ChatGPT is a new AI chatbot developed to deliver the answers to people’s questions in an easy to understand, natural language format. It can provide detailed essays on a wide variety of topics, based on internet searches, that adhere relatively well to expected syntax and writing styles (Sinha, Burd, and DuPreez 2023). The problem, however, is that not only is it unable to effectively differentiate accurate from inaccurate information, ChatGPT delivers its answers in a confident and unnuanced way that removes critical cues (such as the source of the information) that might allow users to calibrate the accuracy or reliability of its answers.

A recent study provided ChatGPT with 100 questions associated with known misinformation topics (Brewster, Arvantis, and Sadeghi 2023). The chatbot generated false narratives for 80% of the topics, significantly exacerbating and promoting conspiracy theories and misinformation on the internet. While the algorithms used by AI such as ChatGPT can presumably improve to better filter false information (and steps have been taken to reduce some of these obvious sources of misinformation), the central challenge will remain that people need key information to help them determine how much confidence to place in the information provided by AI systems.

Typically, people rely on a number of cues to determine how much confidence to place in information provided by a system including the source of information, conditions underlying the generation of the data, its recency, the presence of incongruent or conflicting data, and noisiness or ambiguities in the data (Endsley and Jones 2012). Good design principles stress the importance of providing insight into these factors in system displays to support human understanding of information confidence (Endsley and Jones 2012).

However, these cues are hidden when the AI relies on natural language for its communications. Further, because the AI is integrating information from a wide variety of sources, the reliability of the aggregated outcome is unknown. Parts of what it provides may be accurate and other parts inaccurate. Conclusions may be only partially justified, and sources may be cherry picked or questionable, making conclusions misleading. Not only are these issues not visible to users, many people may mistakenly be overconfident in the output of AI systems (Garcia et al. 2022; Gillespie et al. 2023; Howard 2020), believing them to be more capable than they really are.

Therefore, while natural language interfaces for AI systems may be viewed as desirable, providing more human-friendly interactions, they also pose a simultaneous problem for assessments of their reliability and trustworthiness. Given that people may have many difficulties in determining when and how much to trust AI systems (Kaplan et al. 2023; Stanton and Jensen 2021), a problem that is exacerbated by misleading naming of AI systems (Forster et al. 2018; Liu et al. 2021; Teoh 2020) and media coverage of AI (Fast and Horvitz 2017; Kelley et al. 2021), addressing this problem can be quite complicated.

Overcoming the ironies of AI – future directions

Support for human interaction and oversight

The key implication of Irony 1 is that although AI systems may be very useful for certain applications, they will not take over functions and tasks completely. Rather AI will need to be able to work with and facilitate human decision making as a key objective in their design (National Academies of Sciences Engineering and Medicine 2021). Due to AI limitations, a central focus of the design of AI systems must be to facilitate effective human decision making when AI systems are involved. As the insertion of AI systems into many different applications appears bound to continue, at varying levels of capability, a new framework is needed for ensuring that these systems address the needs of the people who must interact with them. An initial set of requirements for more human-centered AI should include the following.

Attribution

AI systems must be explicitly identified as bots. Further any product of a bot (e.g. generative language, videos, images, recommendations) must be labelled as such, along with the source of its products. This will help people to better calibrate their expectations and assumptions with regard to the veracity of its outputs. This will partially address Irony 5, improving people’s ability to assess the trustworthiness of AI outputs. Given the problems of potential over-trust, terms including ‘intelligent’ and ‘autonomous’ are highly discouraged due to the promotion of unrealistic expectations of system capabilities.

Explainability

AI systems must be equipped with explainability features that allow people who interact with it to understand the systems’ capabilities and limitations for performance (including what factors it does or does not consider in its assessments), partially addressing Irony 2. Because AI can be both opaque and changeable, developing and implementing effective AI explanation systems is important for helping people to develop accurate mental models of the AI. The benefits of AI explainability have been demonstrated in several studies (Bass, Baumgart, and Shepley 2013; Oduor and Wiebe 2008; Paleja et al. 2021).

Considerable work has been directed towards the challenge of creating explainable AI (Samek et al. 2019). Approaches include attempting to create surrogate models, examining how the model behaves under various perturbations, propagating model predictions, and providing meta-explanations though methods such as heatmaps to show what features the model is utilising (Samek and Müller 2019). For example, research has generated understandable rules from ML systems (Huang and Endsley 1997), language predicates from graph models (Kazhdan et al. 2023), and linguistic insights into classification criteria (Saranti et al. 2022).

These approaches have often been more useful for model developers than end users, however. People still need to manually construct an understanding of the AI over time based on AI matching features used in different cases, for example, so as to determine how it might perform in different situations. They must work to generate their own mental models of the system’s capabilities, which can vary significantly across people, even with such aids (Druce et al. 2021). Druce et al. (2021) also found that to be effective a) explanations need to be tailored to the situational circumstances in which the behaviour occurs, b) many AI systems are very brittle and their logic is not easily amenable to explanation, c) methods for providing causal explanations of AI behaviour are not straightforward, and d) people are prone to adopt highly inaccurate mental models of how AI systems work and these models are highly resistant to correction. Effective methods for AI explainability still need further development.

Developing an understanding of causality has been shown to be critical for the development of appropriate trust in AI systems (Shin 2021). AI must be capable of communicating to the user why it makes particular recommendations or takes actions in each case, tailored to the needs of the user. Sanneman and Shah (2022) provide a number of recommendations for improving AI explainability towards this end. Human-centered approaches to AI explanations need to consider the capabilities of the human receiver (e.g. expertise, bandwidth, prior knowledge and assumptions) as well as provide effective methods for explanation delivery.

Although explainability features can be useful for developing mental models of how the AI works and its potential limitations, the use of explainability features is limited for many applications where people need to make real-time decisions and do not have the time to delve into complex explanations. An overload situation can be easily created in which people do not have time to dig into the factors behind each AI action or recommendation.

In dynamic situations, real-time system transparency conveyed through the user interface is more likely to be useful (Endsley 2023). Real-time transparency also has the advantage of being able to reflect AI capability changes associated with ML that people may not have kept up with, and for providing an understanding of AI’s performance and capabilities within the current context.

Transparency

Effective mechanisms must be provided for helping people to understand the capabilities and limitations of AI systems so that they can adopt effective strategies for use, which will partially address Ironies 2 and 3. Two extensive meta-analyses on this topic showed that significant benefits of AI transparency have been demonstrated with regard to reducing OOTL, and for improving both SA and trust (van de Merwe, Mallam, and Nazir 2022; Wickens et al. 2022).

The information needed to support AI transparency is shown in Table 1 in terms of three factors: (a) Taskwork SA – SA of the world and system that is needed to support task performance, (b) Agent SA – SA of the AI that is needed to provide effective oversight, and (c) Teamwork SA – SA needed for coordinating the functions and behaviours of the human and AI (Endsley 2023). AI systems must be transparent to users in terms of: the information it perceives, its integrated interpretations of that information, its projections of future events, its current state or mode, its goals and currently assigned functions, its ability to perform tasks in the current situation and upcoming situations, how well it is currently performing tasks, and its projected actions. Further the source of information, reliability of information and confidence in assessments should be explicitly communicated to users. Endsley (2023) provides an approach to detailing the specific requirements for AI applications, and developing effective formats for providing transparency.

Table 1. Shared SA requirements in human-AI teams—defining AI transparency needs (from Endsley 2023).

SA Area	Supports	Shared SA Needs
Taskwork SA SA of World and System	Task Performance	Perception (Level 1)
		Comprehension (Level 2)
		Projection (Level 3)
Agent SA SA of Team-mate and Self	Agent Oversight, Intervention & Backup	State/Mode
		SA Sufficiency (meta-awareness)
		Ability to Perform Tasks
		Task Status & Performance
		Projected Actions
		Impact of Actions on Others
Teamwork SA SA for Team Integration	Team Coordination & Cooperation	Goals
		Functions
		Decisions, Strategies & Plans

Exposing bias

AI systems must be capable of exposing their own biases and learning limitations to users, addressing Irony 4. AI biases are largely the result of limitations in the generalisability of training sets, or non-causal patterns that may be present in those data sets. AI developers must carefully determine what biases and generalisability limitations exist, and document them for users. Approaches for detecting and mitigating AI bias are being explored (Kiyasseh et al. 2023; Mazijn et al. 2022; Srivastava and Rossi 2019). Because, over time, there is always the possibility that the AI will be used in situations that it is not suited for, it is important that the limits and biases of a given system be clearly indicated to end-users. In addition, a level of meta-awareness of its own limitations needs to be developed within AI systems.

Meaningful control

People must be provided with meaningful control over AI systems (Boardman and Butcher 2019; Cavalcante Siebert et al. 2023), to include the ability maintain situation awareness when overseeing the performance of systems controlled by AI, and the ability to take corrective actions over the AI in a timely manner, partially addressing Irony 3. The challenges of lowered cognitive engagement and human attention are fundamental problems. Even with effective AI transparency, if people are reliant on the AI and thus directing their attention elsewhere, they may not be cognisant of problems demanding their intervention, particularly in continuous control tasks such as driving or flying. Mechanisms for increased engagement (e.g. though divisions of functions that retain meaningful roles for people and allow them to maintain expertise) will be important as well as the ability to effect meaningful control of operations (Boardman and Butcher 2019). AI systems will need to develop a level of meta-awareness (Kounev et al. 2017) that allows them to understand when situations are beyond their capabilities and they should ask people for help. Effective mechanisms for transferring control to people, that takes into account the need for them to rebuild their SA of the situation, must be provided.

Training and skill retention

To addresses the previously identified irony of deskilling, methods for ensuring that people effectively develop and retain cognitive skills required for performance (both alone and in conjunction with the AI) must be developed and implemented (Casner and Hutchins 2019; Paranjape et al. 2019). Relatively little research has been directed at the effect AI may have on the development and maintenance of needed skills. Without the ability to perform complex cognitive tasks, the ability of people to detect insufficient behaviour by AI, or to assume those tasks themselves, is severely compromised. Training should encompass both the requirement to acquire the fundamental knowledge and skills associated with manual performance, as well as the ability to understand and work with the AI. Further, due to the changing nature of ML based AI, refresher training will need to be instituted, so that changes in AI capabilities over time can be learned.

Joint testing of Human-AI systems

The safety and reliability of AI systems when working in conjunction with people must be carefully tested before they are deployed in any safety critical environment. The capability of the AI for allowing people to understand it, and to detect and overcome problems in realistic conditions should be determined. Traditional metrics such as system usability and suitability for purpose should be augmented by more detailed metrics such as the SA and trust of the users when working with the AI system.

This testing should also determine unforeseen behaviours of the AI and the ability of people to detect and react appropriately, as well as unintended consequences (e.g. behavioural changes that occur when the AI is present). A recent study points to the need to consider how human behaviour, the environment and AI blind spots can align to create new types of performance challenges which should be the focus of joint human-AI testing (National Academies of Sciences Engineering and Medicine 2021).

Because of the changing nature of AI, it is also envisioned that a lifecycle approach to testing will be needed, wherein the impact of changes to the AI system over time will need to be tested and verified with human users prior to each deployment. Metrics such as AI sustainability and auditability become important in practice, speaking to the need for people to be able to identify, track and manage system changes as the context of operations changes (National Academies of Sciences Engineering and Medicine 2021). The ability of AI implementations to meet its objectives when working in realistic conditions with actual users is critical. An effective capability for the co-development and testing of the AI system as a part of a joint human-AI team is critical to its success.

Improvement of AI capabilities

Lastly, the AI community continues to work to improve the capabilities of AI systems, making them more ‘intelligent’, addressing Irony 1. In addition to improving their robustness and ability to work in real-world situations, this includes efforts to integrate an understanding of causality into AI approaches (Pearl 2009; Pelivani and Cico 2021; Schölkopf et al. 2021), which would significantly improve their ability to do situational predictions for improved decision making. Work is also ongoing to develop AI systems with ‘self-awareness’ (Chatila et al. 2018; Kounev et al. 2017). This includes the ability to determine when the AI system is dealing with situations that are outside of its boundaries of operations. Such systems would be highly beneficial for avoiding some of the issues associated with AI bias and brittleness.

In addition, there is significant interest in developing AI systems with a higher level of capability for teaming with humans (Liang et al. 2019; McNeese et al. 2018; National Academies of Sciences Engineering and Medicine 2021). In conceiving of the AI as a team-mate with its human counterparts, the importance of mutual collaboration and assistance in adapting to demands becomes emphasised. Rather than providing only one-way information, for the human and AI to team will require substantially more two-way information, with the AI capable of monitoring human performance, aligning its self with human goals, and proactively providing information or assistive tasking for example. Human-AI teaming would require a shared understanding of goals, plans and actions, and shared SA (Endsley 2017b, 2023; U. S. Air Force 2015). Both improved communications mechanisms as well as improved displays and sensing capabilities will likely be required to achieve this goal.

Conclusions

AI has received significant market penetration in a variety of areas, even though its current capabilities remain limited. Rapid technical advances, however, are likely, and along with it increased challenges for human performance when overseeing and interacting with AI systems. Addressing these existing and new ironies of AI will be essential for achieving the expected benefits of AI and averting the long history of problems that are much too likely.

In the long term, significant changes to the job market are possible (with some jobs perhaps being made obsolete). However, in the foreseeable future, the prognosis is for AI to become incorporated into many existing jobs and human activities. In order to avoid the potential negative downsides of the AI ironies described, an active effort to create human-centered AI systems that incorporate the capabilities and approaches described here is essential.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.