To automate or not to automate: advocating the ‘cliff-edge’ principle

Abstract

We reflect briefly on the last forty years or so of ergonomics and human factors research in automation, observing that many of the issues being discussed today are the same as all those decades ago. In this paper, we explicate one of the key arguments regarding the application of automation in complex safety-critical domains, which proposes restraining the capabilities of automation technology until it is able to fully and completely take over the task at hand. We call this the ‘cliff-edge’ principle of automation design. Instead, we espouse a use for the technology in a more problem-driven, human-centred way. These are not entirely new ideas and such a philosophy is already gaining traction in ergonomics and human factors. The point is that in a given system, tasks should be controlled either by human or by automation; anything in between only causes problems for system performance.

Practitioner summary

Human factors problems with automation have been with us for over forty years, and have changed little in that time. This brief review shows a groundswell of opinion that points to what we call the cliff-edge automation principle – restraining the full capabilities of technology until it is ready to fully and completely take over the task. This approach improves human performance in the system by keeping the person in the loop and in control. Researchers and practitioners in ergonomics and human factors should continue to push this message to the designers and manufacturers of automated systems.

Keywords:

• Automation
• human-centred design
• human performance
• safety

The (old) problems with automation

Readers of this journal will be aware that ergonomists and human factors (E/HF) specialists have been debating the relative merits of automation for at least forty years (e.g. Bainbridge 1983). Those same readers will also be aware that, for the most part, the core arguments from an E/HF perspective have changed little during that time.

It was in the 1970s that the information technology revolution led to predictions that automation would reduce complexity in our lives, enhance productivity, and improve quality of life through relieving us of tasks that are difficult, time-consuming, or subject to error (Boff 2006). In reality, we learned that such technology actually served to increase complexity in many tasks and, rather than reducing errors, it has in fact introduced new types of error (Billings 1991), a classic example being mode errors (Sarter and Woods 1995; Stanton and Salmon 2009; Stanton, Dunoyer, and Leatherland 2011). Anyone working in the arena of automation human factors would recognise these issues as equally applicable to the latest automated (complex) systems; because many such systems still rely on humans playing some role in the system, problems typically arise at the human-automation interface (Stanton and Marsden 1996).

One of the central pillars in the response of human factors to the challenges of automation has been to design the system around the user: their needs and wants, their capabilities and limitations. Originally, this took a Fitts-esque approach to allocation of function (see e.g. Hancock 2019), offering tasks to the human or machine depending on which they were better at performing. Subsequently, numerous taxonomies emerged describing levels of automation (e.g. Endsley and Kaber 1999; Parasuraman, Sheridan, and Wickens 2000; Sheridan and Verplank 1978): from fully human control, through various combinations of partial automation, to fully automated control. But these taxonomies were largely descriptive and did not really tell us how or why a given task should be allocated to human or machine. Accordingly, the current zeitgeist focuses on human-automation teaming, using the technology to support, rather than replace, the human operator (e.g. Dekker 2004; Schutte 1999; Young and Stanton 2002; Young, Stanton, and Harris 2007; see also de Winter, Petermeijer, and Abbink 2022). In doing so, the human is kept ‘in the loop’, improving their performance and thereby the performance of the overall human-automation system.

The situation is even more nuanced than that, because some tasks can be automated with no overt impact on the human operator, depending on the level of information processing involved (cf. Parasuraman, Sheridan, and Wickens 2000; Young, Stanton, and Harris 2007). Tasks that require little or no conscious processing, or those that require a response beyond the limits of human performance, may even benefit from automation. Examples of such include traction or stability control systems in cars, while some fast jet aircraft contain such complex avionics that it is impossible to fly them without the aid of a computer. What these represent, though, are part-task automation, which is fine if it is limited to the relevant sub-task. The problems arise when automation purports to assume a more consciously controlled task – and, worse still, only some of that task (i.e. partial automation).

Despite a weight of E/HF research literature demonstrating the consequences of this approach, in practice the technology-centred philosophy of ‘automate everything we can’ has prevailed. The problem with this philosophy lies in those two latter words, because ‘automate everything we can’ rarely equates to ‘automate everything’. Limitations in technology in many areas means that there often remains some role for the human in otherwise automated systems, being on hand to take over as and when the situation goes beyond the capabilities of the automation. Problems then arise with almost – but not perfectly – reliable automation, because human operators come to rely on it, and are then not prepared to take over when required (and then typically get the blame for doing so; see Chu and Liu, 2023).

As we have already noted, these problems are not new. Lisanne Bainbridge’s seminal 1983 paper on the ironies of automation was undoubtedly a watershed moment for the discipline; many of us working in this field have been inspired by this work and continue to cite it even to this day. One of Bainbridge’s core arguments is that the human operator is often only accommodated in automated systems to pick up the ‘tasks left over’ that the automation cannot do, leaving a patchwork quilt of tasks which does not offer meaningful work that the operator can make sense of. Crucially, one of these tasks is being ready to take over should the automation fail or should the situation go outside its design specification. Later, in 1987, James Reason expanded on this in describing the ‘catch-22′ of human supervisory control, in that operators can only take over if they are themselves practiced at the task, which is impossible if it is automated most of the time. Ironically, it is exactly in those scenarios when a human operator could use some automated support. The catch, then, is that automation is most effective when it is least required, and vice-versa. Picking up on this, Don Norman’s 1990 paper told us that the problem is not automation itself, just the fact that it is at an intermediate level of intelligence: it can cope with many things, but not everything. Bringing this brief review full circle, the problem then is that humans are only retained in the system to step in when the automation can no longer cope – a task for which people are ‘magnificently disqualified’ (Hancock 2014). Humans are not best suited to supervising an automated system for any length of time, as decades of research into vigilance have shown (e.g. Mackworth 1948). ‘[I]f you build systems where people are rarely required to respond, they will rarely respond when required’ (Hancock 2014, 453).

The cliff-edge automation principle

Norman (1990) suggested that for automation to be useful, we should either improve it or remove it entirely. Rather than automating where we can, then, the human factors approach considers whether we should automate at all (cf. Parasuraman 1987). In our recent book on driving automation (Young and Stanton 2023), we developed this argument into a philosophy of automation design that we termed ‘cliff-edge automation’. Following the principle that the user should retain a meaningful role in the system, remaining in active control of the task to optimise their performance, we argued that automation should be restrained until such a time as it can fully take over the task with perfect reliability (which, in many cases, will not be any time soon). This is what we meant by the cliff-edge: rather than a gradual slide towards full automation, transitioning through the problematic intermediate levels as the technology evolves and becomes more capable, we should instead hold back until it is possible to jump straight to full automation (see Figure 1 for a conceptual illustration).

Figure 1. Conceptual illustration of the human-centred ‘cliff-edge’ principle through notional levels of automation (from 0 – no automation, to 5 – full automation). Rather than implementing each level of automation when it becomes technologically possible, thus reducing human involvement in a stepwise fashion (dotted line), we should maintain human control until we can step straight to full automation (solid line) (Source: Young and Stanton 2023).

Schematic graph showing notional level of human involvement on the y-axis against level of automation on the x-axis. A dotted line shows the technology-centred approach, with level of human involvement stepping down as level of automation increases, while a solid line shows the human-centred approach, which retains a high level of human involvement until a sudden ‘cliff-edge’ drop when automation reaches its highest level.

There is widespread support in human factors for this cliff-edge type of philosophy in limiting the full functionality of automation (e.g. de Winter, Petermeijer, and Abbink 2022; Hancock 2017; Kaber and Endsley 2004; Mueller, Reagan, and Cicchino 2021; Schutte 1999). In the automotive domain, Norman (2015) made a convincing argument that almost-full automation is most problematic, as drivers come to rely on it and so struggle to take over control when needed (see also Noy, Shinar, and Horrey 2018). If the system appears to be very able, but is actually imperfect, drivers might overtrust it and think it can do more than it is actually capable of (Banks, Eriksson, et al. 2018; Banks, Plant, and Stanton 2018; Ljung Aust 2020). Lee and See (2004) suggested that, in some circumstances, a simpler but less capable automation may be better than a more complex but less trustable version. Indeed, it was argued a very long time ago by Wiener and Curry (1980) that aviation automation had already passed its optimal point. So full automation in itself is not the problem; the difficulties arise in transitioning through intermediate levels of automation to get there (Norman 2015).

We are starting to see shades of this approach in current practice, as some aviation models are indeed predicated on going straight to full autonomy, because this is seen as less complex than transitioning through a human-in-the-loop model (CIEHF 2020). Even some of the trailblazers of automated vehicles have considered skipping partial automation levels, in which the human might need to intervene, and instead pushing straight on to fully automated vehicles (Noy, Shinar, and Horrey 2018). The point is, we either hand over control fully and completely, or else keep a human in the driving seat – literally and metaphorically (cf. Banks and Stanton 2016; Stanton et al. 2020).

A way forward

But it is likely to be several decades before automation technology is good enough to fully and completely take over a task with no human fallback option – certainly as far as a complex system such as driving is concerned, anyway. So what happens in the meantime? Do we play Luddite and try to block the implementation of these systems? The answer, of course, is no. We said a long time ago (Young and Stanton 1997) that we are not technophobes and we would not be able to stop the tide of technology even if we wanted to. More to the point, there is a convincing argument¹ made for driving automation that we should embrace the technology as soon as possible anyway, even if it is not perfect, because although accidents of automation will happen, there will not be as many as those caused by (distracted) human drivers. There is some statistical merit to this argument, as some models suggest that even if automated vehicles are only slightly safer than human performance, hundreds of thousands of lives can be saved over a period of 15–30 years (Kalra and Groves 2017).

Such statistics may be debated as to whether they account for the shortfalls in benefits of automation as a consequence of the human factors problems (cf. Stanton and Marsden 1996). Nevertheless, as Seppelt and Victor (2016) point out, this still puts us in a bind where if we automate then human performance gets worse, but if we do not automate then we negate any potential benefits of automation for road safety (cf. Norman 2015). We have been focusing in this paper on the effect of automation on the human, because in many automated systems the human is the crucial last link in the chain to maintain overall system performance. But, in fact, we should be considering the big picture of that overall performance, and the trade-offs between automated and human control. So how do we reconcile our cliff-edge principle with this argument?

The answer is to exploit the same technology that makes automation possible (that is, the sensors, processors and algorithms), but rather than using it to remove tasks from the human operator, which effectively neglects their needs, instead provide them with additional information or support where they need it most. This problem-driven approach maintains that we should not use technology for its own sake – even reverting to a low-technology solution, or possibly not using the full potential of the automation, in favour of optimising human performance (Hancock et al. 1996; Owens, Helmers, and Sivak 1993). Whatever solution is offered, it should address a need on the part of the human.

Take adaptive cruise control (ACC) as an example. As we know, the argument for automation is often based on evidence from errors or accidents (e.g. Broughton and Markey 1996); the case for ACC was in part based on the fact that over a quarter of all road traffic collisions are due to rear-end collisions (Gilling 1997). If we break this down, it follows that drivers have some difficulty perceiving relative speed in a car-following situation. But do we really need a technological solution for that problem, or would a low-tech approach suffice? Perhaps we should instead build on the success of centre high-mounted brake lights (Farmer 1996) and improve the perception of vehicle rear-ends?

We said earlier that we are not technophobic, so we could alternatively use the same technology to different ends, providing the driver with information to support the task that they normally do, rather than taking over that task for them (Billings 1991; Wiener and Curry 1980). This approach can reduce workload while maintaining situation awareness (Selcon and Taylor 1991; Selcon, Taylor, and Shadrake 1992) as well as negating any concerns about resuming control in the event of failure (Wickens et al. 2015), as the driver maintains control of the task and the system simply provides them with extra information. As such, this would avoid many of the problems of automation associated with being out of the loop.

Applying this to the ACC example, the system’s sensors could be used to provide drivers with advice and/or warnings about the speed (relative or actual) of, or headway from, the lead vehicle. Instead of assuming longitudinal control for them, this would support their judgement of time-to-collision (Stanton and Young 2005), which is a complex perceptual judgement especially difficult for inexperienced drivers (e.g. Cavallo and Laurent 1988). Using technology for information acquisition and analysis exploits the computing power to take care of calculating and integrating information (Seong and Bisantz 2008), supporting drivers’ judgement and thereby adding value to the human-automation relationship. In a similar way, Navarro, Mars, and Young (2011) suggested vision enhancement as an example of perception support, for the 75% of crashes on rural roads that are a result of poor markings of lanes or road edges (That said, Stanton and Pinto (2000) cautioned that behavioural compensation by drivers could negate any safety benefits of vision enhancement).

The kinds of solutions offered above are generic and are in line with the consensus towards human-centred support rather than technology-centred automated replacement (Young and Stanton 2002), fostering human strengths while compensating for their weaknesses (Grote et al. 1995). Much of this can be achieved through the interface display, without necessarily ‘automating’ in the traditional sense (cf. Endsley 1987), as improved sensor and display technology have shifted trends in display design from providing data towards supporting problem-solving and decision-making (Borst, Flach, and Ellerbroek 2015). Similarly, Endsley (2017) stated that automation at earlier stages of information processing (i.e. information acquisition) is more beneficial for situation awareness than at action selection or implementation (see also Stanton et al. 2017; Wickens et al. 2015), arguing that we should automate only where necessary and at the lowest possible level.

In other words, it is better to use technology to support users’ perception than to replace control or make their decisions for them (Stanton et al. 2001). Given that people will still be involved in the control loop for some time to come, they should actually be in control rather than passively supervising (cf. Billings 1991). If that means holding off on full automation until the technology is capable enough, then so be it.

Conclusion

In reflecting on the last 40 years or so of E/HF research in automation, we have highlighted that many of the same human factors problems with automation continue to afflict us. Despite decades of research effort in our discipline, the technology-centred, incremental advance towards full automation still prevails. Just as researchers and practitioners in ergonomics and human factors must defend the role of the human in automated systems (see Chu and Liu, 2023), we must also continue to push our messaging that the technology-centred approach is beset with all manner of human factors concerns. These concerns lie mostly in that challenging middle ground on the levels of automation taxonomies, where the automation can do much of the task, but still relies on a human as fallback. In this respect we agree with Norman (1990), that automation should be improved or removed, and recommend that the cliff-edge automation principle should be adopted as a solution. Notwithstanding technology-centred claims to the contrary, the human factors problems only disappear when fully automated systems that require no human oversight can assume all of the task in all situations.

The aim of this paper was to collate what is becoming a consensus opinion in E/HF research about the application of automation in complex systems, which we have labelled the ‘cliff-edge’ principle, and to present it as a possible way forward. Whilst we have drawn heavily on driving automation examples to illustrate our point, we suggest that the principles are not necessarily limited to the automotive domain. Indeed, many of the issues are rooted in lessons learned from aviation automation (e.g. Billings 1991; Wiener and Curry 1980). A notable case in point is the tragic loss of Air France flight 447 on 1 June 2009, which fundamentally resulted from a mismatch in understanding of the situation between the flight crew and the aircraft’s automated systems (see e.g. Salmon, Walker, and Stanton 2016, for a discussion). There is, undoubtedly, debate to be had about the relative merits of being in manual control for extended periods (such as a long-haul flight). Furthermore, a key distinction between aviation and driving is the level of training for those controlling the machines, which is of course much more rigorous for pilots than for drivers. Whilst such training may serve to offset some of the performance degradations seen when using automated systems (e.g. Young and Clynick 2005), there is still a good argument for designing out such problems in the first place.

But cases such as Air France flight 447 show us that the age-old problems of automation have not gone away, even in aviation, hence the calls for a move towards full autonomy (CIEHF 2020) as per the cliff-edge principle. With more automation entering into more complex, safety-critical domains, it behoves us in the discipline of human factors to continue researching, developing and promoting solutions.

We have put forward the cliff-edge principle as one such solution. Many questions remain about the contexts and circumstances under which it may be applied; these are outside the scope of this brief review and we hope that, instead, they be the subject of further debate and research on this topic.

Until then, we believe a more problem-driven use of technology can help to exploit its benefits while allowing people to remain in the control loop. But we are not entirely anti-automation; in fact, as a closing thought we would return to the observation that for simpler, non-safety-critical tasks, it does have a place. In these tasks, though, there is little need for a human in the loop anyway. Take domestic appliances as an example: once the dishwasher or washing machine door is closed, we can forget about it until it has finished its cycle and it is time to unload. In a similar way, there is a role for automated safety systems that sit quietly in the background until they are needed, without interfering with the task being performed by the human (cf. Young, Stanton, and Harris 2007). Note, though, that these cases have effectively followed the cliff-edge principle anyway, bypassing any reliance on human involvement. Our concerns lie with the complex tasks that cannot be fully automated and still need human supervisory input. It could be said that such systems are not genuinely automated anyway – and so, we might ask, why bother at all?

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes

1 Put forward by Professor Don Norman in a 2015 blog post at: https://jnd.org/automatic_cars_or_distracted_drivers_we_need_automation_sooner_not_later/ (accessed 11 May 2022).