Resilience engineering for highly automated driving, autonomous vehicles, and urban robotics: wizards and shepherds in hybrid societies

Abstract

Introducing automated vehicles into road traffic and robots into urban public environments puts artificial agents in space-sharing conflicts with humans. Humans anticipate, avoid, and smoothly resolve space-sharing conflicts by implicit and explicit communication enabled by uniquely human shared intentionality. Smooth coordination among artificial, partly automated, teleoperated, and human agents needs to be supported by coactive design optimizing observability, predictability, and directability to foster robustness of operation. Available humans should be employed by ‘wizard interfaces’ to support automation, for example, in recognizing intentions, in anticipating difficulties, and in predicting for foresighted coordination. Human capabilities are not just helpful for efficiency and robustness but necessary to cope with exceptional events and failures. For quickly removing blocks and resuming operation, available humans should be employed and authorized by measures of ‘shepherd raising’. Because non-human agents are at a disadvantage in public environments and potentially block operation for extended intervals if they are bullied or denied help, benevolence and cooperation are essential in hybrid societies. Cooperation in public environments is higher in functional societies and can be fostered towards artificial agents in particular by coactive design increasing trust, by influencing mindless and mindful anthropomorphism, and further determinants of reciprocal altruism.

RELEVANCE STATEMENT

Embodied technologies moving as artificial agents in public environments entail challenges of coordination with human agents. Artificial agents in public environments are partially or temporarily operated by humans and should ideally function smoothly as partners in joint activities. From the perspective of Ergonomics and Human Factors, the necessity and benefits of employing humans for robustness of operation and handling exceptional events are highlighted. A range of measures for gaining and supporting human contributions are suggested.

Keywords:

• Space-sharing conflicts
• human support to automation
• coactive design
• graceful extensibility
• fostering cooperation in public environments

Introduction

Highly automated cars and autonomous vehicles moving in traffic, on sidewalks, in hallways and through other public and semi-public spaces will cope with more and more situations without a supervising human intervening or taking over control. The prevailing programmatic statements for automation in road traffic and urban robotics picture vehicles and robots that can navigate safely and perform, for example, transport, delivery, or street cleaning in public environments for extended periods of time without human monitoring or support (e.g. ACEA, European Automobile Manufacturers Association 2019; Christensen et al. 2021; While, Marvin, and Kovacic 2021). These artificial bodies are intended to become capable of coordinating with humans in at times densely populated public environments, for example, as vehicles on streets or as delivery robots on sidewalks. While performing their tasks, they shall function as agents in the joint activity of coordinated space sharing among humans and human-driven vehicles without recourse to human capabilities. What are these human capabilities that technology allegedly becomes independent of? Humans steering vehicles or moving in public environments cooperatively coordinate their actions with copresent humans (Markkula et al. 2020) and flexibly adapt even to unforeseen events (Earl 2017) in traffic and in less regulated, crowded public environments. Human capabilities for flexible behaviour adaptation (Prinz, Beisert, and Herwig 2013), joint action (Sebanz and Knoblich 2021), communication (Clark 1996; Wachsmuth, Lenzen, and Knoblich 2008), and cooperative problem solving (Tomasello 2019) enable humans to often quickly resolve less common, novel, and unforeseen situations. Cooperative coordination and ad-hoc problem solving thus are the fundament of resilience in road traffic and other forms of dynamic public life in the sense of robustness against common disturbances and ‘graceful extensibility when surprise challenges boundaries’ (Woods 2015, 5; see also Woods 2018). The prevailing agenda for highly automated driving and urban robotics implies to introduce into public environments moving technological artifacts that lack these human capabilities and to leave them to their own resources for extended periods of time. This paper invites to ponder consequences of involving ‘unaccompanied’ non-human agents in dynamic public environments, explicates human foresighted cooperative coordination and flexible problem solving in managing space-sharing conflicts, infers the necessity to employ and support human capabilities in order to preserve resilience (Hollnagel, Woods, and Leveson 2006), and argues to follow principles of coactive system design with a widened focus and attention to context characteristics endangering or fostering cooperation. It can be considered an integrative review of theoretical analyses and research on automation and automated driving as well as resilience engineering and human cooperation that derives and substantiates design principles. The main line of argument is summarized in Table 1.

Table 1. Line of argument encompassing theoretical analysis, main inferences, and predictions.

Type	Argument
Analysis	Robustness of operation and overcoming exceptional disturbances (graceful extensibility) are necessary for successful automation
Analysis	Public environments are social task contexts, in which smooth operation is ensured by cooperative agents engaged in joint activity with shared intentionality
Inference	In public environments, space-sharing conflicts between highly automated cars, autonomous vehicles, and urban robots with humans and human-driven vehicles need to be avoided or quickly resolved for robustness of operation
Inference and Prediction	Human capabilities can be employed to avoid (wizard interfaces) and need to be supported (coactive design) to quickly resolve space-sharing conflicts
Inference and Prediction	Human capabilities need to be and can be employed (shepherd raising) to overcome exceptional disturbances
Inference and Prediction	In public environments, cooperation of humans will be less reliable with artificial agents than with humans
Inference	Cooperation towards artificial agents in joint activity and for help in overcoming exceptional disturbances needs to be fostered

Well-known tools for analysing human-automation interaction are theories of supervisory control that originated for task contexts remote from humans such as controlling vehicles on the moon and undersea robots, but are currently also applied to the task context of public environments (Sheridan 2021), for example, simplified as levels of driving automation. Vehicles and robots under supervisory control pursue human-set goals by closing control loops and providing feedback to humans for monitoring and eventual intervention. For physical task contexts, this may capture human automation interaction sufficiently, but public environments are social task contexts. It is not only the operator who monitors, needs information, and should be able to direct and intervene. Rather, less and less closely (tele-)operator-supervised highly automated vehicles and robots moving in populated public environments need to comprehensibly and responsively engage in the joint activity of managing space-sharing conflicts.

Humans anticipating, avoiding, and managing space-sharing conflicts in joint activity

Each location can be occupied by just one agent at the same time, thus, differing intended directions and differing speeds potentially result in space-sharing conflicts. Human traffic participants and pedestrians on sidewalks most of the time achieve smooth coordination to keep traffic flowing and to slow each other down as little as possible. Noticeable conflicts occur if agents have to determine who goes first, for instance, if a vehicle aims to move into an occupied lane or a pedestrian walking on a crowded sidewalk needs to avoid an obstacle. Humans cope with such common disturbances by usually cooperative coordination and thus ensure resilience in the sense of robustness (Woods 2015). Space-sharing conflicts are resolved by implicit and explicit communication among human agents capable of adaptive coordination in joint activity (e.g. Domeyer et al. 2022; Imbsweiler et al. 2018).

A very useful taxonomy of space-sharing conflicts and interactive behaviour for resolving them in road traffic that is also applicable to (urban) robotics more generally has recently been presented by Markkula et al. (2020), who discern as five prototypical space-sharing conflicts obstructed path, merging paths (e.g. lane change), crossing paths, unconstrained head-on paths, and constrained head-on paths (e.g. circumventing an obstacle during oncoming traffic). Markkula et al. present a table of interactive behaviours that have been empirically observed and state: ‘It would seem that, at least in theory, any purpose achieved by a behaviour of a human vehicle driver could be achieved by a HAV [highly automated vehicle] exhibiting a similar or suitably adapted behaviour (e.g. replacing eye contact with eHMI [external human-machine-interface])’ (2020, 733). But – and this is a main point of the theoretical analysis and design considerations of automation in social contexts presented here – replacing the behaviour is not the critical part. While there may be replacements of human behaviour that cooperative humans could interpret similarly (Dey et al. 2020; Domeyer, Lee, and Toyoda 2020; Domeyer et al. 2022; Hensch et al. 2021; Rasouli and Tsotsos 2020; Rettenmaier and Bengler 2021), just being able to show such a behaviour is not enough to achieve what humans achieve in foresighted cooperative coordination when avoiding and managing space-sharing conflicts, particularly in non-standard scenarios (Chater, Zeitoun, and Melkonyan 2022; Straub and Schaefer 2019, Chater et al. 2018). The behaviour needs to be chosen appropriately for cooperative coordination and shown at the right time.

Humans anticipate space-sharing conflicts and assess capabilities, intentional, attentional, motivational, and epistemic states of other agents (Anzellotti and Young 2020; Grahn et al. 2020; Straub and Schaefer 2019; Thornton and Tamir 2021). They communicate to resolve uncertainty in prediction. For example, of 929 verbalizations of drivers asked to comment aloud about any observations in surrounding traffic on a multi-lane ring road around Athens, 360 were observations of social behaviour (Portouli, Nathanael, and Marmaras 2014). Of these, 90 referred to anticipations of others’ intent. Intentions can be gleaned from behaviour that is not just responding to road geometry and surrounding traffic. For instance, ‘The driver behind me speeds up and approaches so I look at him in case he wishes to change lane, to keep an eye on him.’ (1799). Examples of communicating intentions by explicit signals are contained in the following verbalization of a driver who does not want to allow another to cut in (a space-sharing conflict of merging paths): ‘I flashed headlights, I did not let him change lane. He had turned on the direction light, this is an indication, not an obligation for me. I move on my lane, he should change lane after I pass.’ (1801).

Humans anticipate space-sharing conflicts also if they are not directly involved, for example, they anticipate that a vehicle in front is going to brake if a vehicle crossing its path has the right-of-way. Such predictions (as predictions of others’ behaviour in general) improve coordination because they allow for adapting early to being indirectly affected. But humans also predict unimpeded flowing traffic as it results from usual prediction-based coordination among humans and adapt to it. Thus, they are surprised when other vehicles break hard without any apparent reason. The reports of accidents with automated vehicles in 2014 to 2019 in the state of California suggest such expectancy violations (Biever, Angell, and Seaman 2020). Out of 115 collisions while driving in automated mode reported in this period, 68 were rear-impact collisions. Partly, these can be explained by an unexpectedly conservative and reluctant driving style of vehicles in automated mode. But instances of sudden hard braking occur also because automated vehicles lack prediction ability and consequently do not adapt early, which reduces time-to-collision in space-sharing conflicts (e.g. Pokorny et al. 2021). For example, human drivers adapt early by decelerating if they predict an imminent lane change of another vehicle about to cut in in front of them (or by changing lanes if possible or by accelerating). Thus, they avoid hard braking. In contrast, without prediction-based adapting, hard braking is inevitable if a vehicle cuts in in front. Eight of the 68 rear-impact collisions fell in the category ‘Struck from rear during violations of its lane by another vehicle’ (Biever, Angell, and Seaman 2020; see also Hancock et al. 2021).

Automation is inferior to humans in predicting and directing behaviour in social contexts. Thus, just responding to requests for intervention as an out-of-the-loop driver or teleoperator without monitoring and anticipatingly responding to the social context impairs coordination and safety. Certain safety problems can be alleviated by increased automation. Yes, humans are not infallible, they suffer lapses of attention, they can be dangerously distracted, they can be drowsy, intoxicated, uncooperative, risk-prone, and even aggressive. Hence, assistance systems in vehicles such as those that monitor and provide warnings, support lane keeping and emergency braking enhance safety for agents in and outside of vehicles (Furlan et al. 2020). Cooperatively connected vehicles and vehicles interfacing with infrastructure may enhance safety even further (e.g. Jang et al. 2020; Noy 2020). However, focusing only on human fallibility belittles how astonishingly safe and efficient humans manage the frequent space-sharing conflicts on the ground at high and at low speed. There are no records of this daily achievement, but it can be roughly quantified by inference. Hale and Heijer (2006) used data about the vehicle kilometers driven per year per type of road in the Netherlands and estimated 6.5 x 10¹¹ encounters/year with other road users (e.g. at junctions, during overtaking, or road crossing aka in space-sharing conflicts). To estimate the rate of accidents per encounter, they calculated with a factor of 10.000 more injury and damage accidents than fatal accidents. The resulting estimated rate of accidents per encounter of 1.5 × 10⁻⁵ gives an idea of how reliably these daily space-sharing conflicts are managed by humans. The probability of human error is usually estimated higher even in simple tasks. For instance, 4 × 10⁻⁴ is an estimate for highly practiced routine tasks with time to correct potential error (Williams and Bell 2016). In encounters, human capabilities buffer against accidents because consequences of risky behaviour and individual error can be avoided or reduced by compensatory behaviour of other participating agents (Houtenbos 2008). Thus, human capabilities ensure resilience day to day against disturbances, but also in coping with exceptional challenges.

As an example, let me briefly narrate a recent event. During rush hour, I noticed an unusual jam while approaching a crossing of a minor two-lane road with a main inner-city street, on which two lanes in each direction enclose tramway rails. This crossing is regulated by traffic lights except for late-night hours, but they were defect at this time of heavy traffic. No police officers were present to direct traffic and police did not arrive during the 10 min or so that I could participatingly observe. Traffic signs gave right of way to the main road, of course. Thus, drivers on the crossing minor road needed to negotiate their way to the other side. Drivers askingly entering from the minor road when meeting sufficiently cooperative ones on the main road led bouts of crossing cars quickly joined by cars in the opposite direction until this crossing traffic was cut off again by traffic on the main road. This description is simplified. Please, imagine in addition cyclists, the tramway, pedestrians, and cars not sticking to lanes on the crossing and turning right or left. You may have experienced or may have seen footage of such traffic. For the involved road users, it was a surprise that challenged the boundaries of the socio-technical system of road traffic and an opportunity to demonstrate human capabilities playing out in graceful extensibility (Woods 2015, 2018) and hence, ensuring resilience. Usually, space-sharing conflicts are less apparently negotiated but proactively resolved by anticipation. Can superior human capabilities for anticipation in social contexts be employed to assist automation?

Humans assisting automation for robustness and Safety - wizard interfaces

An attentive passenger in an automated vehicle (i.e. the driver of a highly automated vehicle in automated mode) will notice developing space-sharing conflicts and could avoid surprising, abrupt manoeuvres if the human prediction could be turned into a directive for the automated vehicle - similar to warnings of passengers to human drivers if they notice a danger that they think the driver might have missed. Pointing out a vehicle in another lane about to cut in can provide such human assistance to automation in an interesting prototype implemented in a driving simulator (Wang et al. 2021; see Figure 1).

Figure 1. Wizard interface for prediction support to automation. The human can point out a vehicle in the surrounding traffic that is about to change lanes so that the highly automated car can include this prediction in its planning algorithm to avoid hard braking (cf. Wang et al. 2021).

The automated vehicle predicts trajectories of surrounding vehicles and chooses manoeuvre plans accordingly, for example, continuing straight with the same speed, decelerating, or changing the lane. The currently chosen manoeuvre is displayed to the driver. Thus, the current manoeuvre could be to keep speed in the same lane, but the human may predict that a vehicle in the lane to the right is going to cut in, for example, because of a slower vehicle in front, a blocked lane, a vehicle about to move into its lane, or merging lanes. This prediction can be fed into the planning algorithm by gazing at the vehicle to the right and clicking a button (or the voice command ‘watch out’). In response to this input, the manoeuvres decelerating and changing to the left lane gain in value and the initial plan is revised to avoid hard braking. In the visual display to the driver, the revised manoeuvre plan is shown. Such an interface to assist automation on the prediction level would preserve the possibility to employ superior human capabilities (instead of claiming to substitute them completely as conveyed in the prevalent definitions of increasing levels of driving automation; see, e.g. Hopkins and Schwanen 2021; Kaber 2018; Kyriakidis et al. 2019; Sawyer et al. 2021). Such an interface is an exemplary ‘wizard interface’ because similar to the experimenter-in-the-loop ‘Wizard of Oz’-method for lending human intelligence to machines (Kelley 1983), human capabilities augment technology. Note, that the human contribution is not necessary but just enabled. This is not a case of take-over or fixed allocation of a critical subtask, also no case of dynamic allocation because technology is not capable of the contribution. Of course, this interface covers only a subset of scenarios in which prediction is important to keep traffic flowing, and it creates benefit only if the driver is attentive. Yet, the leading idea of humans forming a joint cognitive system with automation in interdependent activity (Johnson and Bradshaw 2021) instead of automation substituting humans is clearly exemplified. Wizard interfaces for assisting automation in prediction increase resilience also in less dynamic scenarios, for instance, if bystanders can communicate their assessment to an urban robot that an obstacle is not going to be removed soon and rerouting should be considered (see Table 2).

Table 2. Humans as supporters for robustness and warding off stalling surprise including wizard interfaces.

Example	Principle
Displaying operation mode − 360° visual display on the exterior of a highly automated car signalling that the car is in the performer role and no human can be expected to quickly respond to explicit communication	Observability
Displaying epistemic state – Visual display on the exterior of a delivery robot that signals whether it ‘knows’ about a temporary or permanent block ahead (red light at a crossing, obstacle, cliff)	Observability
Displaying ‘attentional’ state – Visible indicator (display or mechanical) of an urban service robot that signals whether it is receptive to speech and gesture of a particular individual in its vicinity	Observability
Displaying planned action – Visual display of the destination of a delivery robot and the planned route	Predictability
Communicating intention – Implicit (slowing down early) and explicit (visual display at standstill) communication of a driverless vehicle’s intention to stand back in a space-sharing conflict	Predictability
Communicating request – Auditory signal and visual display or mechanical indicator (movement) of the direction that a delivery robot requests to proceed in on a blocked sidewalk	Predictability
Interface for human support to automation – Driver of a highly automated car in the supporter role can feed diverging predictions such as an imminent lane change of a surrounding vehicle into the car’s planning algorithm (‘watch out’-example in the main text and Figure 1; Wang et al. 2021)	Wizard interface (Directability)
Interface for human support to automation – Passengers of driverless vehicles or bystanders close to an urban robot can feed their assessment into a planning algorithm that a block ahead is not going to dissolve soon and an alternative route should be chosen	Wizard interface (Directability)
Interface for humans to indicate failure or need for assistance – Button for passengers of a driverless vehicle to call for teleoperation or mobile number on a delivery robot for bystanders to report a failure	Directability

Note. Examples of humans as supporters are drivers of highly automated vehicles, passengers of driverless vehicles, teleoperators, traffic participants, and bystanders.

Coactive design of joint cognitive systems for resilient joint activity

Supporting and enabling contributions of human agents in joint activity with automation in a social context (e.g. drivers, teleoperators, pedestrians on sidewalks) can be pursued by implementing the principles of mutual observability, predictability, and directability put forward in the coactive design method and its interdependence analysis (Johnson et al. 2014; Johnson and Bradshaw 2021; see also Christoffersen and Woods 2002; Klein et al. 2005). Coactive design strives for optimizing performance and resilience by identifying and supporting necessary contributions of individual agents to subtasks of a joint activity for a particular role assignment (performer, supporter), but also by identifying, enabling, and supporting potential contributions (as by wizard interfaces) and alternative role assignments.

Joint activities are constituted by interdependencies of agents’ behaviour. Contributions of an agent to a joint activity often depend on coordination with another agent (or multiple other agents) for which mutual observability, predictability, and directability (OPD) need to be fulfilled in an appropriate manner. Examples of OPD are provided in Table 2 and Table 3, succinct definitions of OPD are provided, for instance, by Johnson and Bradshaw:

Table 3. Humans as performers and supporters for graceful extensibility including shepherd raising.

Example	Principle
Decision support system for failure management – for teleoperators, service technicians, trained volunteers, or bystanders	OPD
Extension of teleoperating power – by drones and dinghy robots on-board, or strategically stationed surveillance cameras, drones, or robots	OPD
Interface for human support to teleoperators – mobile audio and video telecommunication between teleoperators and passengers, trained and certified volunteers, or bystanders	OPD
Displaying situational need for assistance – by movement characteristics and visual displays conveying unease and arousal comprehended via mindless anthropomorphism or zoomorphism (e.g. Singh and Young 2013)	Shepherd raising
Interface for requesting help from humans on site – Visual and auditory display to assess, get the attention, and interrupt passers-by and to politely address passengers and bystanders	Shepherd raising
Displaying trustworthiness and signs of community – Visual display of features, signs, logos, or seals of trusted and respected organizations or communities informing mindful anthropomorphism	Shepherd raising
Interface for assessing and authorizing helpers – automated on driverless vehicles and urban robots or for teleoperators to check capability and benevolence, to explain liability and rewards, and to provide access to control	Shepherd raising
Rewarding help – Issuing thanks and gestures of gratitude, benefits, or monetary compensation	Shepherd raising
Voluntary certification – Provide training for volunteers completed by a registered certification with or without membership in an organization	Shepherd raising
Interface for guiding – Sensors and control algorithms for locking to human helpers as guides around obstacles or through a group or crowd (‘follow me’, e.g. Stein et al. 2016)	Directability
Interface for direct control – Handles or touch- and pressure-sensitive surfaces to be directed without or with prior ‘relaxation’ of basic collision avoidance (Lupetti, Bendor, and Giaccardi 2019)	Directability
Emergency-stop control – Salient and accessible for passengers of driverless vehicles, passers-by, and bystanders	Directability
Controls for resuming, resetting, restarting – Accessible without or with proof of basic certification for passengers of driverless vehicles, passers-by, and bystanders	Directability
Fostering willingness to help – Campaign for partnership with embodied artificial agents in public environments	Shepherd raising
Fostering community and trust – Transparency, fairness, responsibility, community service, and sponsoring	Shepherd raising
Foster friendly, safe, and functional public environments – Social equality, public service, participation, consensus building, regulation, law enforcement, livable cities	Shepherd raising

Note. OPD: Observability, Predictability, Directability. Examples of humans as performers are teleoperators, service technicians, registered and trained members of voluntary services, drivers of highly automated vehicles, passengers of driverless vehicles, traffic participants, and bystanders as human helpers on site.

1. Observability refers to how clearly pertinent aspects of one’s status—as well as one’s knowledge of the team, task, and environment—are observable to others. … The complement to making one’s own status observable to others is being able to observe status. 2. Predictability refers to how clearly one’s intentions can be discerned by others and used to predict future actions, states, or situations. The complement to sharing one’s own intentions with others is being able to predict the actions of others. It requires being able to receive and understand information about the intentions of others, to be able to predict future states, and to take those future states into account when making decisions. 3. Directability refers to one’s ability to be directed and influenced by others. The complement is to be able to direct or influence the behavior of others. (2021, 132)

Interdependence analysis discerns strong and weak interdependence relationships. For instance, a human agent will usually determine and set (at least approximately) the navigation goal for a highly automated vehicle. This human input is a necessary contribution fulfilling a strong interdependence. Exploiting weak interdependence relationships by enabling and supporting potential (i.e. not required) contributions of an agent to joint activity can improve regular performance and safety as illustrated, for instance, by driver assistance systems and the ‘watch out’-example of a wizard interface for human assistance to technology (Figure 1). The wizard interface enabling human anticipation support exploits a weak interdependence relationship and exemplifies all three main principles. It is observable to the human that the highly automated vehicle is in automated driving mode and which surrounding vehicles it has detected. After the human has pointed out the vehicle in the right lane ahead, it is observable that its predicted lane change is included in the planning algorithm. The automated vehicle’s behaviour is predictable because its considered manoeuvres and the currently chosen manoeuvre are displayed. Its behaviour is directable because the human input is included in its manoeuvre planning. Thus, using the potential of weak interdependence relationships by designing OPD appropriate to human requirements and attentional resources can keep human capabilities operative in joint cognitive systems for performance and safety. This improves resilience in the sense of robustness against common disturbances. Furthermore, optimizing observability (transparency) and predictability supports situation awareness and all three principles are important to consider for building trust in automation and calibrating trust through experience (Chiou and Lee 2021; Omeiza et al. 2022; Zhang, Tian, and Duffy 2023).

Employing and supporting human contributions instead of shutting them out does not only improve standard operation of automated vehicles among human agents, it also allows for human problem solving and cooperation to cope with surprising challenges such as the failure of traffic lights during rush hour. Shutting human contributions out reduces resilience. The proponents of joint cognitive systems theory have very clearly elaborated why this is the case and how disregarding the actual procedures and practices in a task domain as well as disregarding the limits of automation regularly leads to assuming that humans could be simply substituted by automation, that is ‘that new technology (with the right capabilities) can be introduced as a simple substitution of machine components for people components … This over-simplification fallacy is so persistent it is best understood as a cultural myth’ (Woods and Hollnagel 2006, 167). The substitution myth (Christoffersen and Woods 2002) misses how tasks are actually performed in a field of practice, it misses human contributions to resilience, and it misses that any change in a joint cognitive system triggers adaptations of human agents. In typical examples (e.g. air traffic, healthcare), this fallacy has been explained by the fact that individuals who propose and pursue such simple substitution (e.g. managers, designers) are ‘at the blunt end’ that is detached from ‘the sharp end’ where the actual work is performed. Interestingly, this is clearly not the case for joint activity in road traffic and on sidewalks that highly automated vehicles and urban robots do and will participate in and with which every proponent of simple substitution should be familiar from participating at the sharp end.

Supporting (tele-)operating humans taking over control for graceful extensibility

For coping with exceptional challenges, that is for resilience in the sense of graceful extensibility, it can be critical to provide the option that humans take over control. All humans present and available – not just drivers and teleoperators – can be considered as a resource and the OPD principles of coactive design are valuable for supporting humans individually in troubleshooting but also joint problem solving. Drivers can take over in highly automated vehicles (Lee et al. 2022), teleoperators are taking over in current concepts for driverless vehicles and urban robotics (Bogdoll et al. 2022; Goodall 2020), further resources of human capabilities to be tapped and supported by system design following OPD with a widened focus are, for instance, passengers of driverless vehicles, registered and qualified volunteers, service personnel, and bystanders.

In July 2021, a law came into force in Germany that allows to operate driverless vehicles on designated sections of the public road network. A supervising human (‘Technische Aufsicht’) who can be remote is required for deciding on manoeuvres suggested by the vehicle or to take over control after the vehicle moved into a state of minimal risk. Such a state is defined as standstill at a safe location with hazard warning lights active. For delivery robots or other urban robots operating autonomously without a monitoring human nearby, typically also the switch to teleoperation is the intended primary solution to resolve deadlocks or failures. Deploying operators or technicians to the vehicle or robot is the option usually pursued when teleoperation does not suffice.

In teleoperation for resuming after a driverless vehicle moved into a state of minimal risk and for resolving deadlocks and failures, the teleoperating humans are hampered compared to if they were on site (Kamaraj, Domeyer, and Lee 2021). The delays and restricted possibilities of gathering information, communicating, and acting from remote limit teleoperators in bringing into action human capabilities. Interfaces designed for OPD can support teleoperators who ideally should quickly assess the situation and then diagnose and intervene to achieve recovery. The challenge for teleoperators as in failure management in general is to determine what is wrong and how to fix it. Decision support systems as troubleshooting aids can provide assistance if failures are anticipated and covered in support systems’ domain models (Banerjee 2021; Honig and Oron-Gilad 2018; Yanco et al. 2015). Exceptional events, however, have to be dealt with without apposite decision support.

The limited possibilities of teleoperators to gather information and to act via interfaces to the vehicle or robot can be extended by mobile technology on board that can be deployed to provide additional perspectives and opportunities to act, for example, drones and dinghy robots. Immobile surveillance cameras and robots stationed strategically distributed are further possibilities of extending teleoperating power (both could be part of a shared infrastructure). Weighing benefits against costs of such technological extensions of teleoperating power may result in relying on mobile human service that will arrive with a certain delay. But often capable human assistance can be expected to be on site already that teleoperators can turn to for help, for instance, passengers, bystanders, and members of professional or voluntary services. Employing human helpers on site and increasing the availability and willingness of human helpers will be referred to as ‘shepherd raising’ in analogy to fundraising (see Table 3).

Employing humans on site for graceful extensibility – shepherd raising

Driverless vehicles that move passengers likely have humans on board. Driverless vehicles and robots operating in urban areas likely are located close to bystanders. Passengers and bystanders can be addressed and directed by teleoperators, for example, for operating cameras and other devices or tools needed for diagnosing and intervening. Passengers and bystanders can remove obstacles, clean sensors, address additional humans, and act in many other ways as assistants with general human capabilities. For such ad-hoc collaboration, both teleoperators and human helpers on site need to be supported by suitable interfaces for telecommunication and operating. An audio and video connection should be available and ideally, this telecommunication will be mobile. Teleoperators need to assess the capabilities and trustworthiness of potential helpers, select helpers, and if applicable authorize helpers for access to controls. Of course, safety, security, privacy, and liability need to be considered and addressed. Motivation, readiness for, and quality of help would be improved by a basic voluntary certification (‘technical first aid’) and (micro-)rewards. A proof of such certification would establish helpers’ trustworthiness towards teleoperators.

Potential human helpers on site are a resource offering human capabilities to achieve graceful extensibility not only in teaming with teleoperators, but also if employing and directing them by human teleoperators is not possible. A failure may interrupt the wireless connection, teleoperators may be unavailable, automation may have reached its limits (see Figure 2) or a failure or operating problem has not been recognized as such by the vehicle or robot and thus, there was no call yet for teleoperation or service. If a failure or problem has been recognized by a vehicle or a robot and teleoperation is not feasible, letting the vehicle or robot address potential helpers on site for assistance is an option to be considered (Banerjee 2021; Brooks et al. 2020; Honig and Oron-Gilad 2021; Srinivasan and Takayama 2016; Weiss et al. 2010). Depending on the scenario, the need for assistance could be signalled to get the attention of potential helpers, or humans recognized in the vicinity could be addressed individually, for example, pedestrians on sidewalks or passengers on board of driverless vehicles. In driverless vehicles, addressing individuals could be more successful with designated seats for passengers willing to help or those with basic voluntary certification. For comparison, think about the recruiting of and instructions for airline passengers on seats next to emergency doors.

Figure 2. Shepherd raising for overcoming limits of automation. A human bystander who is registered as a certified volunteer is politely addressed by a zoomorphically designed robotic mower for assistance in crossing a road via direct guidance using a touch-sensitive area. The human may earn micro-rewards.

In addressing potential helpers, politeness increases the probability of receiving help (Kopka and Krause 2021; Oliveira et al. 2021; Srinivasan and Takayama 2016). Cues presenting technology as a social actor are processed without much thought (mindless anthropomorphism or zoomorphism) or with awareness (Hortensius and Cross 2018; Lombard and Xu 2021; Nass and Moon 2000). The appearance and (interaction) behaviour of a vehicle or robot should be designed such that humans are motivated to help whether they respond without much thought or with higher awareness of design purposes and awareness of whom they are actually helping if they agree to assist (but see Young 2021). For example, design features of sound, light signals, and movement (Cha et al. 2018; Pezzulo et al. 2019) can communicate arousal and ‘unease’ directly appealing to empathy (Singh and Young 2013) and in addition a logo can communicate the well-known and positively connotated operating company or respected organization (Celmer, Branaghan, and Chiou 2018; Riegelsberger, Sasse, and McCarthy 2005). Regarding safety, security, privacy, and liability, similar precautions should be taken as when helpers are involved by teleoperators. Again, a kind of certification and rewards would increase trustworthiness, readiness, and motivation. Ad-hoc helpers not directed by teleoperators need to be supported in diagnosing and intervening by interfaces designed for OPD as well (Honig and Oron-Gilad 2018). If applicable, decision support systems can be implemented (Banerjee 2021).

For dealing with exceptional events, helpers should be able to quickly gain direct and intuitive control so that they can move the vehicle or robot, for example, to resolve deadlocks, mitigate danger, or circumvent an obstruction. A joystick, a handle, or a touch- and pressure-sensitive surface area could provide such control for a mode, in which the vehicle or robot immediately responds to directing movements (cf. H-mode, Flemisch et al. 2014; see also Abbink, Mulder, and Boer 2012; Flemisch et al. 2019; Navarro 2019). Collision avoidance and other safety measures may stay activated by default, but when envisioning the range of exceptional events that may occur, providing the option to deactivate such features that limit interventions seems worth considering (‘relax button’).

As a minimum, a control for an emergency stop is usually provided for passengers. A clearly identifiable and easily accessible emergency stop seems sensible also for bystanders, who by their human capabilities can notice and anticipate problems that a driverless vehicle or robot may miss or notice too late (Mutlu and Forlizzi 2008). Of course, providing such an emergency-stop control incurs the danger of misuse. This challenge of unfriendly or even malevolent behaviour will be addressed in the section on fostering human cooperation with artificial agents. Further minimal intervention controls for passengers and bystanders would be an option to contact teleoperators or service personnel for reporting a problem (which could lead into teaming with teleoperators) and a control for resuming operation (e.g. after an emergency stop). Furthermore, simple controls for reset or restart could be helpful after interventions that may offset model assumptions or cause incongruencies in a vehicle’s or robot’s software at runtime. This condensed enumeration of possibilities is intended to widen the focus as appropriate for a social task context and to point out more and less obvious available resilience-strengthening human contributions and how they may be tapped and supported. However, humans can also be less helpful, uncooperative, and even dangerous in public environments.

Artificial agents face obligations and disadvantages in public environments

Human agents in public environments are exposed, protected by norms and regulation, and have obligations. It is at present unsure whether artificial agents lacking human capabilities will be accepted in public environments and treated cooperatively in space-sharing, yet, reflecting on potentially critical factors may help to foster acceptance and consequently resilience.

Driverless vehicles in road traffic and robots for urban environments operate in public environments that are regulated by access control and norms for behaviour of agents. Agents lacking adult human capabilities are excluded or confined. Animals that could bring about impediment or danger as autonomous agents lacking human capabilities are removed from or strictly confined in civilized public environments. Humans and other agents lacking adult human capabilities, young children or human-owned animals, for instance, are supposed to be monitored, confined, and assisted as needed while they roam public environments. The basic rules of public safety as part of public order have been succinctly summarized by Goffman (1963, 23): ‘In going about their separate businesses, individuals—especially strangers—are not allowed to do any physical injury to one another, to block the way of one another, to assault one another sexually, or to constitute a source of disease contagion.’ Blocking the way unintentionally happens easily while entering space-sharing conflicts without human capabilities.

Introducing non-human, artificial autonomous agents in public environments is a challenge to public order and resilience not only because they lack human capabilities, but also because they may not be treated as equal to human agents in space-sharing conflicts with humans. Non-human agents are not necessarily accepted as a partner in cooperation (Caruana and McArthur 2019; Silver et al. 2021) and are not granted favors and rights equally to humans (Karpus et al. 2021; Liu et al. 2020). In road traffic, there are regulated and unregulated space-sharing conflicts. Even in regulated space-sharing conflicts, automated vehicles and urban robots are at a potentially stalling disadvantage. For instance, if a pedestrian intends to cross the road at a location not constructed or marked for crossing and an automated vehicle approaches on the road, the vehicle obviously has the right-of-way, but the pedestrian may step on the road more likely than if the vehicle was human-driven. Defensiveness and reliable collision avoidance put automated vehicles and robots at a disadvantage in the chicken game of space-sharing conflicts (Chater et al. 2018), which – even if exploited infrequently – endangers resilience in the sense of robustness and safety (Liu et al. 2020; Millard-Ball 2018; Thompson et al. 2020). Hiding whether a vehicle is driven by a human or in automated mode would be no fix because already testing for a defensive response by pretending to cross would stall automated vehicles as it stalls robots. Presumably, the disadvantage of automated vehicles is aggravated if humans are neither on board nor in the traffic behind because then, stalling the vehicle does not block the way of a human.

A pessimistic vision inferred from this disadvantage of reliable collision avoidance pictures dedicated, fenced off lanes, zones, or networks for automated vehicles. Such a spatial division of humans and automation would safeguard smooth functioning in a then non-social task context, but would not deliver many of the envisioned benefits of automated vehicles and urban robotics. Alternatively, artificial agents and humans may share urban zones of pedestrian primacy, in which vehicles and robots could operate only slowly during busy hours. During hours, in which few people are on the street, there would be less potentially stalling space-sharing conflicts, but in sparsely populated public environments, driverless vehicles and robots are in higher danger of falling prey to pranks, crime, and vandalism than humans. This is another aspect, in which artificial agents cannot be expected to be treated equal to humans by humans. Less scruple against property damage than physical injury needs to be factored in if artificial autonomous agents are to be introduced into public environments.

Fostering human cooperation with artificial agents in public environments

Public environments can be safe and pleasant, but they can also be dangerous and demanding. The same geographical location can at times be the one or the other depending, for instance, on physical infrastructure and its integrity, the functionality of a government, administration, and legal system, the weather, or simply the time of day and the people present in private and professional roles. In general, friendly, safe, and functional public environments are less challenging for artificial agents. In those environments, it seems not overly optimistic to expect security by monitoring through the public eye, sufficiently spread readiness for cooperation with artificial agents, and sufficiently spread readiness of potential human helpers to contribute their capabilities for resilience of socio-technical systems.

Humans as a species have a unique readiness and competence for flexible ad-hoc cooperation (Tomasello 2019), but this powerful flexibility entails that they can decide for and against cooperation (Misyak et al. 2014). Readiness for cooperating and help despite individual short-term costs does vary between individuals (Darley and Batson 1973; Kopka and Krause 2021), between societies and cultures (Fehr and Fischbacher 2003), and between places and situations (e.g. Currano et al. 2018). Humans are likely behaving cooperatively and helpful if such behaviour fulfils social norms of the cultural common ground that they are (more or less) sharing, partly because of internalized beliefs, values, and norms, and partly because social norms are codified, enforced, and violations are formally or informally sanctioned.

Importantly, human cooperation is malleable and fostered, yet not bound by rules that in some contexts exist as explicit expressions of social norms and conventions (e.g. road traffic). Adult humans sharing cultural common ground (Clark 1996; see also Domeyer, Lee, and Toyoda 2020; Klein et al. 2005; Veissière et al. 2020) expect each other to act responsibly, they expect each other not to stick to rules if doing so would not be sensible.

Being responsible means being socially or morally cooperative with others: when others produce a good reason to change your actions, you do so. And indeed, as the reason-giving process is internalized, others – especially those with whom one shares in cultural common ground the group’s beliefs, values, and norms – expect you to anticipate the reasons that ‘we’ might give for particular beliefs or actions and thus do the right thing of your own accord. (Tomasello 2019, 330)

Shared intentionality encompassing joint intentionality among individuals in joint activity but also collective intentionality is playing out when humans flexibly, creatively, and proactively contribute to robustness and graceful extensibility of socio-technical systems. The socio-cultural context, situation-specific and structural features of public environments, perceived community with owners of vehicles or robots, interface design (including OPD), and the trade-off of costs and benefits for individuals need to be considered when assessing how likely humans will choose cooperation and contribute human capabilities to resilience (or diminish the benefits of automation by exploiting defensiveness, performing vandalism, or committing self-enriching crime). All of these factors that influence behaviour can be modified and all are related to aspects of trust – trust of individuals encountering automated vehicles and robots in public environments, and trust of owners and operators (Chiou and Lee 2021; Riegelsberger, Sasse, and McCarthy 2005). Cultural common ground and community strengthen reciprocal altruism in networks of agents. For example, in the Federal Republic of Germany, an extensive campaign was run from 1971 to 1973 to foster reciprocal altruism in road traffic with the slogan ‘Hallo Partner – danke schön’ to turn around rising fatality rates. The campaign is considered to have been successful also in influencing public opinion in favour of regulations (codified social norms) that improved traffic safety (e.g. speed limit on rural roads, alcohol limit). There was a similar campaign in the German Democratic Republic with the slogan ‘Aufmerksam, rücksichtsvoll, diszipliniert – ich bin dabei’ (attentive, considerate, disciplined – count me in).

The willingness to perform costly actions without immediate personal benefit (or to forego taking benefit at the cost of others) increases with trustful expectations that one is going to benefit in the long run from similar behaviour of others (Fehr and Fischbacher 2003; Ostrom and Walker 2003). Fostering such willingness and expectations is an integral part of resilience engineering because sustained adaptability of a network of agents is dependent on the willingness of agents to adapt to assist other agents if those operate close to their limits (at risk of saturation) or need help. Perceived community (McMillan and Chavis 1986) increases such willingness. Presumably, a communal street-cleaning robot will receive bystander assistance to get back on track quicker (by more individuals) than a food delivery robot of a company providing commercial service. Tolerance against being blocked by such urban robots and willingness to step back in space-sharing conflicts with them will also differ (Abrams et al. 2021). Thus, commercial service providers will have to invest more in perceived community and rewards than communal service providers to earn respect and to receive assistance for driverless vehicles and robots deployed in public environments.

Resilience engineering for feasability and acceptance of artificial agents in public environments

Currently, there are hardly any highly automated vehicles and urban robots operating in public environments. Hence, some requisite imagination (Westrum 1993) is required to gauge the challenges to resilience if the proportion of artificial agents without human capabilities increases in networks of agents sharing limited space on the ground. With regard to resilience of operations, the density of activities and the degree of spatial restriction are critical. Obstructions and frictions create repercussions more likely, more quickly, and at a larger scale if space is shared by a larger number of navigating agents, and if there is less free space for resolving space-sharing conflicts or to evade obstructions. In terms of cognitive systems engineering, this constitutes ‘tight coupling’ producing more complex disturbance chains following disruptions, intensifying demands of anomaly response, and challenging coordination and resilience (Woods and Hollnagel 2006). With tight coupling, unexpected events more likely dynamically escalate cognitive and coordination demands.

Artificial agents will be more often surprised and stalled than human agents, they will more often encounter situations they cannot handle and will move into a stationary state of minimal risk, they will also sometimes stop operating because of failures. All such instances potentially impede or completely block other agents. Blocked artificial agents are less able than humans to resume. Consider the example of a blocked single lane divided from the lane of oncoming traffic by a solid line. Human agents would (responsibly) disregard the rule against crossing this line to circumvent the block or to turn around for a detour on their own initiative or they would follow directions by humans (e.g. police) to do so. Because stalling is annoying, impeding, and dangerous, and because blocking in busy scenarios can quickly spiral into hard to resolve jams, feasibility and acceptance of driverless vehicles and urban robotics will critically depend on minimizing stalling and blocking, and on quickly resolving and removing blocking. Hence, human capabilities need to be employed for robustness and graceful extensibility instead of attempting to completely substitute them. Otherwise, higher numbers of artificial agents in public environments are just not feasible. There are expensive detours to be saved by picturing and acknowledging this inconvenient truth, and avoiding that ‘developers miss higher demand situations when design processes remain distant and disconnected from the actual demands of the field of practice’ (Woods and Hollnagel 2006, 135).

Conclusion

While automated vehicles and urban robots are introduced into public environments, there will be trial and adjustment. A lot of futile trials, delays in innovation, and expenses could be avoided, if it would be acknowledged and taken into account first of all, that human capabilities currently ensure smooth coordination and resilience in public environments and that completely substituting them by automation reduces resilience. Second, humans are capable of joint activity also with artificial agents and can be supported in contributing their capabilities for resilience. Third, humans can decide for and against cooperation, and consequently, there is risk in entrusting humans with tasks and room for manoeuvre in socio-technical systems and in putting reliably cautious and defensive agents in space-sharing conflicts with humans. Fourth, instead of shutting humans out and reducing resilience, these risks should be reduced and humans should be employed for resilience by proven and novel measures for increasing benevolence and cooperation on individual and societal levels. Only then, highly automated vehicles and robots sharing space with humans in public environments are feasible and potential benefits of increased safety, reliability, and efficiency can be realised.

Disclosure statement

The author reports there are no competing interests to declare.

Additional information

Funding

This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Grant Project-ID 416228727 – SFB 1410.

Notes on contributors

Georg Jahn

Georg Jahn obtained his PhD in Psychology from the University of Regensburg, Germany, in 2001. He held an assistant professorship for Cognitive Psychology at the University of Greifswald and an associate professorship for Engineering Psychology and Cognitive Ergonomics at the University of Lübeck. Currently, he is Professor for Geropsychology and Cognition at Chemnitz University of Technology where he was Spokesperson of the Collaborative Research Centre Hybrid Societies: Humans Interacting with Embodied Technologies.