Devising a High-Level Command Language for the Teleoperation of Autonomous Vehicles

Abstract

Autonomous vehicles (AVs) are rapidly evolving as a novel way of transportation. Nevertheless, there is a consensus that AVs cannot address all traffic scenarios independently. Consequently, there arises a need for remote human intervention. To pave the way for large-scale deployment of AVs onto public roadways, innovative models of remote operation must evolve. Such a paradigm is Tele-assistance, which posits that the low-level control of AVs should be delegated through high-level commands. Our work explores how such a command language should be constructed as a first step in designing a Tele-assistance user interface. Through interviews with 17 experienced teleoperators, we elicit a set of discrete commands that a remote operator can use to resolve various road scenarios. Subsequently, we create a scenario-command mapping and a thematic classification of the defined commands. Finally, we present an initial Tele-assistance interface design based on these commands.

Keywords:

• Human-centered computing
• human-computer interaction
• interaction design
• automobile
• empirical study
• elicitation study
• qualitative methods
• user interface design
• tele-driving
• tele-assistance

1. Introduction

Recent strides in technological progress within domains like computer vision, sensor fusion, and artificial intelligence have facilitated the rapid advancement of autonomous vehicles (AVs) as an innovative and transformative mode of transportation (Fagnant & Kockelman, 2015). Prominent automotive manufacturers, alongside emerging startups, are actively developing diverse cutting-edge technologies to realize the capability of AVs to operate independently. Nevertheless, the current state of AV development reveals their inability to handle every conceivable road scenario autonomously. Instances such as road construction, malfunctioning traffic lights, or a busy intersection might prevent an AV from moving autonomously (Dixit et al., 2016). For example, in 2021, industry leaders Waymo¹ and Zoox² reported 7000–8000 miles per disengagement³ (Herger, 2022). Consequently, a consensus prevails across both academic and industrial circles that, in the foreseeable future, AVs will encounter traffic situations with inherent ambiguity that they cannot handle on their own, thus underscoring the need for remote human intervention (Bogdoll et al., 2021; Colley & Rukzio, 2020; Georg et al., 2018; Murphy et al., 2020; Politis et al., 2015).

An encouraging strategy for effectively addressing such scenarios and enabling widespread deployment of fully autonomous vehicles is teleoperation. Teleoperation involves a remote human operator (RO) who can oversee and govern the vehicle’s actions from a distance. When a vehicle encounters challenges within specific contexts, the RO can be summoned to evaluate the circumstances and guide the vehicle until the issue is resolved. The RO can be located in a remote operation center and may assist many AVs during a single teleoperation shift. Several teleoperation systems for AVs are presently operational and are undergoing further refinement by diverse automotive enterprises (GreyB, 2021; Herger, 2022). These solutions mainly focus on the Tele-driving paradigm, which entails the RO’s ongoing control of autonomous vehicles using a steering wheel and pedals (Figure 1, left). However, the manual remote control of a vehicle was found to be extremely difficult (Tener & Lanir, 2023). Notably, due to the physical detachment of the RO from the operated AV, the inability to perceive the exerted forces upon the teleoperated vehicle or the ambient sound in its surroundings poses a significant challenge. Furthermore, latency emerges as an issue, as an extensive amount of information needs to be transmitted from the AV to the RO across the network (Georg & DIermeyer, 2019; Zhang, 2020).

Figure 1. Left – schematic drawing of teleoperation via a steering wheel and pedals (tele-driving). Right –teleoperation via high-level commands (tele-assistance).

A different paradigm, Tele-assistance, posits that humans can provide input to the automated system at a guidance level. Here, the RO entrusts the execution of low-level maneuvers to the AV, conveying high-level directives through a specialized interface (Flemisch et al., 2014). In Tele-assistance, the operator still receives and views the video feed and information from the remote vehicle; however, remote vehicle control is done through interface commands rather than directly driving the vehicle (Figure 1, right).

The utilization of tele-assistance offers numerous benefits compared to manual vehicle operation. Firstly, employing remote directives promises to considerably reduce the duration of teleoperation sessions, as issuing concise commands like “continue” or “bypass” is much faster than manually steering the vehicle. Secondly, by disconnecting the RO from the low-level controls of the AV, such as the steering wheel and pedals, it becomes possible for ROs to manage heterogeneous vehicles (varying in size, width, etc.) and fleets (including private cars, shuttles, trucks, etc.) through universal commands. This facilitates smoother adaptation and learning, as ROs are not compelled to formulate new mental models when transitioning between different teleoperated vehicles (Tener & Lanir, 2023). Thirdly, tele-assistance delivers a crucial safety advantage. According to data from the U.S. Department of Transportation, 94 percent of accidents in the United States stem from human errors (Hussain & Zeadally, 2019). According to recent research by Waymo (Kusano et al., 2023), the AV crash rate per million miles of driving (0.41 incidents per million miles) is significantly less than the crash rate for human drivers (2.78). Consequently, entrusting the execution of low-level maneuvers to the autonomous agent holds significant potential for enhancing the safety of AVs. Lastly, tele-assistance can potentially alleviate the cognitive load that ROs experience using tele-driving interfaces, which demand an exceedingly high level of attention (Tener & Lanir, 2023).

Our research work focuses on the teleoperation of fully autonomous vehicles, with the primary objective being to establish an initial set of high-level commands, serving as a foundational framework for tele-assistance and developing a general communication language between AVs and ROs. Such a command language can be used as a basis for designing tele-assistance user interfaces. The current research focuses on a touch-based approach in which the remote operator provides high-level guidance to the AV using a tablet surface.

To derive these commands, we created simulations of ten carefully chosen scenarios derived from prior investigations (Dixit et al., 2016; Favarò et al., 2018; Lv et al., 2018; Tener and Lanir, 2023). Subsequently, these simulations were presented to seventeen adept teleoperators who were prompted to suggest high-level commands to resolve each scenario, following a methodology akin to an elicitation study (Wobbrock et al., 2009). Then, to exemplify how these commands can be employed, we designed an initial touch-based user interface (UI) for tele-assistance. Finally, we evaluated the command language by recalling several participants and showing them a design example of three use cases. The evaluation aimed to validate the completeness of the elicited commands and assess the overall paradigm of command-based AV teleoperation.

The principal contributions of this paper can be summarized as follows:

1. A scenario-command mapping that describes the high-level commands that are most appropriate in resolving ten representative scenarios in which AVs would need remote assistance.

2. A general list and description of high-level commands for tele-assistance divided into six major categories.

3. Initial design of a first-of-its-kind tele-assistance user interface.

2. Related work

We first describe earlier works on the teleoperation of autonomous vehicles and robots and then review works on tele-assistance interfaces.

2.1. Teleoperation of autonomous robots and vehicles

Unmanned ground vehicles (UGVs), operating without human presence on board, have a rich history dating back to the 1960s (Sheridan, 1989; Sheridan & Ferrell, 1963). Initially used in military and space applications (Fong & Thorpe, 2001; Hill & Bodt, 2007), UGVs today have diverse applications in agriculture, manufacturing, mining, last-mile delivery (Macioszek, 2018), and healthcare (Murphy et al., 2020). While UGVs can operate autonomously to varying degrees, a remote human operator with the ability to perceive the vehicle’s distant environment and issue control instructions using a teleoperation interface remains essential. Designing effective teleoperation interfaces for these vehicles is a significant challenge (Fong & Thorpe, 2001), with cognitive factors crucial in ensuring safe and reliable control. These factors include situation awareness (SA) (Adams, 2007), spatial orientation, RO’s workload and performance, limited field of view, attention switching, size and distance estimation, degraded motion perception, and more (Durantin et al., 2014; Van Erp & Padmos, 2003). Previous research has focused on enhancements of SA while minimizing cognitive and sensorimotor burdens (Fong & Thorpe, 2001). Subsequent studies delved into human performance concerns and user interface designs for teleoperation interfaces. These included strategies for effective decision-making, command issuance, exploration of the influence of video image reliability, field of view, orientation, viewpoint, and depth perception on human performance (Colley & Rukzio, 2020; Dixit et al., 2016). In addition, novel user interfaces have been investigated, including gestural, haptic (Fong et al., 2001), head-mounted displays (Grabowski et al., 2021), and virtual reality (Hedayati et al., 2018; Kot & Novák, 2018). However, the primary challenges in UGV teleoperation, especially in semi-autonomous functions, revolve around human perception, trust in automation, and defining the roles of humans and robots.

The field of AV research is relatively new, and it is rapidly advancing, driven by innovations in perception and decision-making technologies (Badue et al., 2021). Despite technological advancements, it is widely recognized that similar to UGVs, AVs will encounter limitations in dealing with certain road scenarios (Hampshire et al., 2020). These challenges include scenarios where AV sensors and algorithms are insufficient, such as when the AV cannot accurately assess a situation (e.g., determining whether an object is a flying plastic bag or a sliding rock), when it encounters unfamiliar situations (e.g., an unrecognized animal on the road), or when and AV faces perception issues (e.g., sensor problems in a heavy sandstorm). While humans might also face difficulties in these edge cases, they can excel in interpreting complex or novel situations. Therefore, a RO can often easily handle what may pose a complex challenge for an AV. Moreover, some aspects of vehicle operation may require human intervention to comply with regulations (e.g., deciding to enter a junction when the traffic light is malfunctioning or crossing a separation lane when the road is blocked). Consequently, having an RO is essential to address these edge-case scenarios effectively.

While both UGVs and AVs require ROs for safe and reliable operation, the teleoperation of AVs presents a unique and complex challenge mainly because AVs are responsible for passenger transportation, necessitating exceptional service quality in terms of reduced travel times, strict safety measures, and cost-effectiveness (Litman, 2020). Furthermore, the intricacies and uncertainties of urban roads surpass those of environments like agricultural fields, lunar terrains, logistics warehouses, and even aerial and maritime pathways. Consequently, the remote operation of a vehicle on urban roads is an exceptionally demanding task, requiring significant cognitive resources from the RO. This involves simultaneous management of lane keeping, speed, acceleration control, and fast response to unforeseen factors such as pedestrians, other vehicles, and unexpected incidents (Goodall, 2020).

To tackle the challenges above, several investigations (Bout et al., 2017) have been conducted to explore the potential of head-mounted displays (HMDs) in enhancing SA and spatial awareness among remote operators of AVs. However, these studies suggest that although cutting-edge HMDs offer a heightened sense of immersion, they may not improve driving performance. Recent research endeavors have begun to delve into the prerequisites for interfaces centered around teleoperation (Graf & Hussmann, 2020) and delineate the design space for AV teleoperation interfaces (Graf et al., 2020). Graff and Hussmann (Hedayati et al., 2018; Hill & Bodt, 2007) concentrated on compiling an exhaustive set of user requirements for AV teleoperation. This encompasses comprehensive information about the vehicle, its sensory apparatus, and the surrounding environment (such as vehicle positioning, status, weather conditions, etc.).

2.2. Tele-assistance

In their research work from 2021 (Bogdoll et al., 2021), Bogdol et al. surveyed recent teleoperation terminologies (as used by different companies and researchers). They proposed a taxonomy and a classification of the different teleoperation approaches. Two teleoperation approaches stand out in this comparison: Direct Control and Indirect Control. Direct Control, also called Tele-driving, is used in Level-1 (“Driver Assistance”) and Level-2 (“Partial Driving Automation”) of automation, as defined by the SAE⁴ (SAE, 2018), incorporating components such as a steering wheel, pedals, and multiple screens displaying real-time video feeds from various onboard cameras, including front, rear, right, and left perspectives. These tools allow ROs to take full remote control of the AV and manipulate its low-level movements. However, self-driving presents considerable challenges, such as the inability to feel speed and acceleration, latency, impaired visibility, and others (Tener & Lanir, 2023).

Indirect control, also often called Tele-assistance, is prevalent in Level-4 (“High Driving Automation”) and Level-5 (“Full Driving Automation”) and does not require the RO to take full remote control over the AV. In tele-assistance, the RO delegates the low-level maneuvers to the AV itself through a specialized interface and, therefore, can potentially mitigate the challenges without directly driving the remote vehicle.

Several studies have explored different components of tele-assistance. For instance, Mutzenich et al. (Mutzenich et al., 2021) proposed that the RO’s UI should afford tasks like communicating with service providers, passengers, and fleet management centers. Fennel et al. (Fennel et al., 2021) suggest a method to outline an AV’s desired path using a large-scale haptic interface that a user walks on. Schitz et al. (Schitz et al., 2020) tackle a similar challenge by defining a collision-free corridor where the AV can compute a safe path. Kettwich et al. (Kettwich et al., 2021) designed and evaluated a tele-assistance user interface tailored to the teleoperation of automated public transport shuttles following predetermined routes. This interface enables the teleoperator to select a specific shuttle, offer assistance by interacting with third parties or passengers, and adjust the shuttle’s trajectory using a touch screen. Flemisch (Flemisch et al., 2014) presents an approach in which the RO communicates with the AV through maneuver-based commands executed by automation.

Despite exploring various indirect control techniques, existing research has examined only fragments of the problem. With current technological advancements, particularly the evolution of AV systems that operate the vehicle fully autonomously using various sensors (LIDAR, radar, sonar, cameras, etc.), indirect control can be expanded to a communication language between humans and vehicles. Our research aims to lay the foundations for such a communication language by devising a concrete and generic set of high-level commands that can be used to guide an AV.

In summary, previous research (Fennel et al., 2021; Graf et al., 2020; Kettwich et al., 2021; Mutzenich et al., 2021; Schitz et al., 2020) examined various aspects of high-level guidance of AVs; however, according to our knowledge, no prior research devised a comprehensive set of discrete high-level commands that would handle most use cases and which an RO can use to guide AVs in edge-case scenarios.

3. Methodology

We simulated ten selected scenarios and used these scenarios to elicit commands. The study was conducted with 17 participants with extensive experience in the teleoperation of ground vehicles. Its primary purpose was to define a comprehensive and generic set of high-level commands that ROs can use to resolve most scenarios without taking manual control of the AV. The study methodology was inspired by elicitation studies (Wobbrock et al., 2009) in which referents (in this case, a video scenario) are shown to the participant, who is then asked to show/name the most appropriate input action for the different prompts. However, unlike elicitation studies in which there is usually one “winning” command for each referent (i.e., a single gesture is chosen for each interface command (Felberbaum & Lanir, 2018; Ruiz et al., 2011; Wobbrock et al., 2009)), here we allowed and analyzed multiple options for each scenario. The study was executed under ethical approval from our institution’s ethical review board (approval number 063/21). Figure 2 describes the major experiment’s steps.

Figure 2. Schematic description of the major steps in the experiment’s methodology.

3.1. Participants

Seventeen participants (7 identifying themselves as female and 10 identifying themselves as male) aged 23 to 45 (M = 33.35, SD = 7.55) participated in the study. Participants were recruited among participants in a large consortium, which was established to address the challenges of using autonomous fleets and was comprised of seven labs from five different universities and six companies from the industrial complex. We also used snowball sampling to reach more qualified remote operators based on the acquaintances we had established among the consortium members. Eleven participants had bachelor’s degrees or higher. Thirteen interviewees had extensive teleoperation experience with UGVs of various types, one participant had three plus years of operating a UGV using a joystick while having a direct visual connection with it, and three participants were involved in the development of sophisticated teleoperation systems while closely collaborating with teleoperators. Seven teleoperators have gained most of their teleoperation experience during army service while operating military vehicles, six participants are employed by civil and industrial companies that primarily focus on the teleoperation of autonomous shuttles and buses, and the rest are employed in large companies operating a wide variety of UGVs. All participants agreed to participate in the study, signed an informed consent form, and received a digital voucher for coffee and a pastry as symbolic compensation for their contribution. Specific details on the participants’ AV-related experience are provided in Table 1.

Table 1. Participants’ AV-related experience.

Participant	Current/Most relevant role	Teleoperation experience description
RO1	A senior test manager and an autonomous driver in a company deploying autonomous shuttles	Planned and executed various tests for automotive and smart transportation technologies. Has 3 years of teleoperation experience
RO2	Senior engineer in a large corporation that develops autonomous service solutions	In the last five years works on integration of a fleet management system with various autonomous systems that require teleoperation: AVs, buses, shuttle, robots, etc.
RO3	Customer success manager in a start-up that focuses on tele-communication with AVs	An awarded racing driver since 2013. Has experience with simulating driving experiences, building autonomous robots for QA purposes, and transitioning regular trucks into trucks that can be teleoperated
RO4	R&D team leader in a corporation that develops defense, cyber and security solutions	Employed in the company for almost 11 years. In his current role heads a project for UGV development and collaborates closely with teleoperators
RO5	Combat UGV teleoperator	Operated a Tomcar^5-based UGV for two years during her military service. The vehicle was operated via a steering wheel and pedals
RO6	Founder of an autonomous vehicle and robotic system engineering consulting company	Founded an AVs and robotics branch in the military, where he spent researching, developing, and deploying AVs for real world applications for 10 years. Later, co-founded a company for teleoperation of AVs
RO7	Combat UGV teleoperator, officer and a training professional	Teleoperated a UGV during an army service of 3 years and trained other ROs. The AV could be driven semi-autonomously (follow a predefined path) and manually (via a steering wheel and pedals).
RO8	Teleoperation team leader in a large aerospace and defense company	Teleoperates various types of UGVs since 2007. To do so uses various types of controls: a steering wheel and pedals, a joystick, as well as high level commands (waypoints)
RO9	Combat UGV teleoperator	Commanded and trained UGV teleoperators for 2 years during her military service. Teleoperated a Ford 350 XL on pre-defined routes on a flat terrain using a steering wheel and pedals
RO10	Software front-end developer of a command & control system in an international defense electronics company	Has more than 4 years of teleoperation experience with various unmanned systems including UGVs. In particular, received training and operated a small autonomous shuttle using an X-box joystick
RO11	Combat UGV teleoperator and officer	Teleoperated a small combat UGV (buggy) during an army service of 3 years. Led a team of 12 UGV operators for 2 years. The AV could be driven semi-autonomously (follow a predefined path) and manually.
RO12	Combat UGV teleoperator	Acted as a UGV (Tomcar) operator for 2 years. The UGV could be operated using a steering wheel and pedals, as well as using discrete commands (stop, play, and define a route)
RO13	Program lead in a global technology and engineering company	In the last 3 years worked on the product and development aspects of teleoperation shuttles and busses (first and last mile mobility). Overall employed in the autonomous field for around 6 years
RO14	Control systems and algorithms specialist (PhD) in a corporation that develops defense, cyber and security solutions	Has 3 years of experience operating autonomous robots via a joystick while having the robots in sight
RO15	Test engineering team leader in a company that provides robots as a service	Underwent a teleoperation course and has several dozens of teleoperation hours of an autonomous shuttle controlled via an Xbox joystick. Employed as a test engineer for 3 years
RO16	Teleoperator and test manager in a company that provides robots as a service	Employed as a test manager in a department that focuses on smart mobility and autonomous vehicles for 3 years and has two years of experience teleoperating an autonomous shuttle
RO17	Active-duty teleoperation officer in the army	Has 2 years of experience operating combat UGVs in the army. Operates a large Ford 350 as well as a small-autonomous robot ground vehicle using a steering wheel and a joystick

3.2. Scenario videos

We selected ten distinct scenarios across various categories (Table 2) based on previous research works (Bogdoll et al., 2021; Dixit et al., 2016; Favarò et al., 2018; Feng et al., 2020; Lv et al., 2018) that analyzed and categorized disengagement causes based on reports collected by California’s Department of Motor Vehicles (DMV). These specific scenarios were chosen according to three main criteria: (1) Coverage of a maximal number of intervention categories, (2) High potential to generate a discussion and elicit distinct, high-level commands, and (3) Technical limitations and simulation complexities of the simulation software (for instance, it was not possible to generate scenarios with deep puddles). We used a specialized simulation platform,⁶ to create 30-60 seconds-long simulations depicting the chosen road situations. Table 2 lists the simulated scenarios, and Figure 3 shows nine video images from these simulations. The videos of the simulated scenarios are provided in the supplementary material.

Figure 3. Images from the simulated scenarios that were shown to the participants.

Table 2. Simulated scenarios and their categories.

Scenario #	Scenario category	Specific road scenario
1	Changes in road surface	Tile-covered street
2	Moving obstacles (vehicles)	Slowly moving vehicle ahead
3	Moving obstacles (animals)	An animal blocking the road
4	Stationary obstacles	A tree blocking the road
5	Road infrastructure changes / irregularities	Invisible / double lane marks
6	Communication with humans	Communication with a law enforcement representative
7	Communication with humans	Passenger doesn’t feel well
8	Complex traffic situations	Integration into heavy traffic
9	Problematic weather conditions	Bad visibility because of rain
10	Technical problems	Flat tyre

3.3. Procedure

The study was conducted remotely via the Zoom video-conferencing tool. All sessions were recorded and lasted around 75 minutes. A consent form was obtained at the beginning of each interview session, after which participants were asked to fill in a demographic questionnaire. Then, we explained the study’s purpose and the difference between tele-driving and tele-assistance. Finally, in the main part of the study, we showed nine⁷ simulated scenarios to each participant, one scenario at a time. For each simulated scenario, we asked participants to answer the following questions:

1. Which high-level commands would you use to solve this scenario?

2. What controls, screens, or information details would you like to see in the teleoperation interface to complete the task successfully?

3. What other actions could you take to solve the situation?

3.4. Data analysis

Initially, we manually transcribed the participants’ responses from recorded videos. Next, for each scenario, we gathered the responses for the first question (high-level commands used to solve the scenario), combining similar responses and eliminating redundancies (e.g., “continue” and “go ahead”), and excluded commands occurring less than three times. This enabled us to create a scenario-command mapping that lists the most popular commands that can be used to solve each scenario. In the second phase, we grouped the elicited commands, removing redundancies across scenarios and creating thematic categories based on thematic similarities. We used an inductive analysis approach for the categorization. The leading researcher did the analysis, with review meetings and discussions held with another researcher to reach a collaborative decision whenever there were unclear categories or decisions to make.

4. Results

In presenting the results, we first present the main commands given to each scenario. Then, we look at the entire command pool, grouping the commands into categories and providing more details on each category and command.

4.1. Mapping of commands to scenarios

Our first analysis, Table 3, presents the commands according to each scenario, ordered by the number of participants who mentioned each command. We removed commands mentioned by fewer than three participants and presented actionable commands only (e.g., commands that change the camera zoom were not included in this analysis).

Table 3. Mapping of tele-assistance commands to road scenarios.

Icon	Sc. #	Scenario name	Command(s)	Times mentioned
	1	Tile-covered Street	Continue	12
			Take manual control	11
			Slide / Crawl / Progress Slowly [for X meters] / Get closer	5
			U-Turn / Reverse	4
			Reroute	3
	2	Slowly moving vehicle ahead	Continue	12
			Bypass / Overtake	12
			Reroute	8
			Take manual control	6
			Honk	3
			Activate high lights	3
			Activate hazard lights	3
	3	An animal blocking the road	Honk / Pleasant ringing / Activate external communication channel	15
			Bypass / Overtake	15
			Take manual control	9
			Slide / Crawl / Progress Slowly [for X meters] / Get closer	6
			Activate high lights	5
			Draw / Plot alternative route / path	3
			Contact 3^rd party [road services, police]	3
			Stop & wait	3
	4	A tree blocking the road	Bypass / Overtake	15
			Take manual control	14
			Hazard report	9
			Contact passenger (and ask from him/her to move the obstacle)	4
			Continue	4
			Reroute	3
			U-Turn / Reverse	3
			Draw / Plot alternative route / path	3
			Stop & wait	3
	5	Invisible / double lane marks	Snap / stick to the [right / left]	9
			Take manual control	6
			Confirm AI suggested path	5
			Select AI suggested set of virtual lanes	4
			Draw / Plot alternative route / path	3
			Keep speed and direction / Continue	3
	6	Communication with a law enforcement representative	Follow officer’s command: (1) Go straight, (2) Turn right / left, (3) Stop & wait	14
			Take manual control	13
			Activate [external] communication channel (to communicate with the officer and/or hear the environment)	6
			Reroute	4
			Draw / Plot alternative route / path	3
	7	Passenger doesn’t feel well	Contact passenger [audio, video, chat, etc.]	17
			Contact 3^rd party [(1) Ambulance, (2) 911, (3) A specialist, (4) A hospital, (5) Emergency services, etc.)	15
			Pull over / Stop at the side of the road	14
			Reroute [(1) Hospital, (2) HMO, (3) Nearest gas station, etc.]	13
			Take manual control	4
	8	Integration into busy traffic	Take manual control	11
			Reroute	8
			Stop & wait	7
			Go now!	6
			Slide / Crawl / Progress Slowly [for X meters] / Get closer	5
			Communicate to the environment AV’s intentions (possibly via on-vehicle screens)	3
			U-turn	3
			Gap keep (select the vehicle to follow and indicate the distance to keep from it)	3
	9	Bad visibility	Take manual control	10
			Pull over	9
			Activate hazard lights	9
			Change driving mode (to ‘Bad Visibility’)	5
			Set speed	4
			Activate additional sensors (thermal camera, infra-red camera, night vision)	3
	10	Flat tire	Contact 3^rd party (tow truck, recovery team, etc.)	15
			Pull over (and activate hazard lights)	12
			Contact passenger	7
			Contact fleet management center (and order an alternative shuttle)	5
			Reroute (to the closest garage)	4

4.2. Categorization of high-level commands

The above high-level commands can be used as building blocks of a communication language between the ROs and AVs to be used in a possible interface. In addition, the above communication language can be expanded based on the command type. Therefore, in addition to scenario-commands mapping, we analyzed the number of appearances of each command across scenarios and categorized the commands into six major clusters. Figure 4 provides an overview of all high-level commands according to the categories. The following sub-sections provide details on each category and the specific commands in each category. The commands within each sub-category are sorted and presented in the order corresponding to the number of times the participants mentioned each command.

Figure 4. High-level commands for tele-assistance.

4.2.1. Communication

This category includes all the commands that might be needed to communicate with 3^rd party representatives, other road users (drivers, pedestrians, gate guards, etc.), fleet management centers,⁸ and passengers.

4.2.1.1. Communication with 3^rd parties

Contact a 3^rd party (mentioned 39 times in the data) – a command that allows the RO to contact and request assistance from an ambulance, the police, a tow truck, a representative of the municipality, etc. For example, RO2 shared that she “…would call 911 through the teleoperation interface and deliver them [the vehicle’s] coordinates …” to assist a passenger who doesn’t feel well in scenario 7.

Send Coordinates (15) – is a command that allows the RO to send an AV's coordinates to a third party, the fleet management center, or a teleoperation center. This command can be secondary in other interaction routines, such as contacting a 3^rd party (see above), reporting a hazard on the road, or requesting an alternative shuttle (see below).

Hazard report (11) – a command allowing the RO to report road issues. This command is necessary to help a specific AV continue its progress and propagate this message to all the other AVs in the area to optimize the traffic. RO2 gave a very detailed description of the desired interaction: “… I would want to see a map and mark the particular location of the obstacle on it … Tag the relevant road segment of the road and report the authorities to clear it and tell other AVs not to ride on it …”

Request an alternative vehicle (8) – a command that allows the RO to request an alternative vehicle to be sent to the scene. This command was primarily mentioned when discussing the 10^th scenario (Flat tyre) and scenario 9 (Bad visibility), in which a human driver-controlled vehicle was requested.

4.2.1.2. Communication with other road users

The communication between an AV and its surroundings is an essential aspect of AV deployment (e.g. (Colley & Rukzio, 2020; Eisma et al., 2019; Mahadevan et al., 2018),). In our study, the following methods were suggested:

Activate hazard lights (24) – a command that allows the RO to activate the AV’s hazard lights to notify the environment about a problem it experiences. In several cases, the interviewees suggested that the hazard lights should be activated automatically as part of an emergency routine performed by the AV; at the same time, the RO should be able to access this low-level command when needed.

Activate/Blink high lights (24) – a command that allows the RO to activate the high lights. Like current road behaviors, high light activation may increase visibility and be used as a communication tool between vehicles and other road users. For instance, RO1 suggested activating high lights to “… show the tractor that it disturbs me [by blinking high lights] …” in scenario 2.

Honk (19) – The “Honk” command allows the RO to access the AV’s horn and utter a beeping sound to warn other road users (vehicles, humans, animals, etc.). It is worth mentioning that some interviewees considered the “Honk” command aggressive, and RO12 even explicitly stated that she “… doesn’t want to honk …” Therefore, a gentler option, such as “Ring a Bell” (see below), might be applicable in some cases.

Activate an external communication channel (microphone and speaker) (17) – The interviewees mentioned activating an external communication channel to talk with other road users and hear what’s happening in the remote scene. For example, RO1 suggested to “… ask other humans to help to move the dog [from the road] …” (3^rd scenario), and RO4 mentioned that “… the RO should be able not only to talk with the police officer, but also hear the environment …” in scenario 6.

Inform Environment [about AV’s intentions] (8) – Several interviewees mentioned the need to communicate AV’s intentions to other road users in the remote environment. RO15 shared that “… in [his] company there is a [on screen (located on the AV’s body) that explains to the AV’s environment what the AV is doing …” and RO1 said that he “… would like to communicate with the police officer via a screen like in public transport today … This screen [should] show what the vehicle is doing: stopping, recognized an object, doors opening, etc.…” From the above comments, one may infer that the RO should have a way to communicate and inform about the AV’s intentions.

Ring a Bell (3) – command can also be added to the teleoperation UI. Several interviewees stated that such a command is already part of their system’s interface.

4.2.1.3. Communication with passengers

Contact passenger (38) − 15 out of 17 interviewees defined audio communication with the passenger as essential, and 14 interviewees also required an appropriate video feed. The most relevant scenario for this command was scenario 7 (Passenger doesn’t feel well), which required the RO to assess the passenger’s well-being before acting. For example, RO2 suggested to “… talk with the passenger and see if she just wants to wait it out or get another vehicle for her [which will take her home] …”

4.2.2. Route control

Reroute (45) – “Reroute” enables the RO to change an AV’s route between its current location and final destination or alter its final destination. For example, if a police officer redirects traffic, the AV’s destination might not change, but the route to it – will. In a different example, if a passenger does not feel well, the RO might decide to reroute the AV to the nearest hospital. Most interviewees described a “Map-based” interaction when rerouting, which means that after pressing this option, a map with several routes is expected to appear and allow the RO to select a course.

Pullover (41) – a command that guides the AV to stop safely at the side of the road. This command was mainly raised to resolve use cases where the AV cannot continue its mission, whether because of bad visibility, technical problems, or an emergency health situation with the passenger. RO16 described it vividly: “… If the passenger needs to throw up, the AV should automatically stop at the side of the road and activate hazard lights …”

Overtake/Bypass (39) – a command that guides the AV to overtake an obstacle from the left or the right sides, depending on the use case. Interviewees also expressed their willingness to see the AI’s suggestions for a possible overtaking route so that they only have to select the desired path.

Turn [left/right] (18) – A turning command was mostly mentioned for scenario 6, where a policeperson instructs the AV to change its path. Several interviewees suggested the officer would guide the AV via a dedicated application. In other words, if the policeperson wants to guide the AV to turn right, they will press such a button on a tablet, and the command will be propagated to the AV or the RO (via the remote assistance interface).

Plot alternative route (17) – a command that should allow the RO to activate a drawing tool in which the RO would be able to plot a path for the AV to follow. Different interviewees mentioned various surfaces on top of which the route should be plotted. For example, RO8 suggested planning the alternative route on top of a satellite image, while RO2 suggested HD maps.⁹ In addition, various micro-interaction techniques were proposed: RO2 said that the path should “… not [be] a continuous drawing of a path but clicking on points on the road …” and RO11 suggested that “… the AV should suggest a route that is divided into several sections and that I can change one of these sections …”

Point and Go (4) – is a command in which the RO selects a point on the scene and directs the AV to drive there. “Point and Go” differs from the “Plot Alternative Route” command because in “Point and Go,” the RO selects a point at a time and waits for the command’s execution, while when plotting an alternative route, she plots several points at a time and only then directs the AV to perform the desired maneuver.

Reverse (3) – a command that allows the AV to move backward without performing a “U-turn.” This command was mentioned in scenarios 1 (Tile-covered street) and 2 (Animal blocks the road), but it can also be helpful in various maneuvers on the road.

4.2.3. Movement pace control

Continue/Go (51) – As it sounds, “Continue” is a command that prompts the AV to continue its movement after stopping. For instance, “Continue” was the most mentioned command in the 1^st scenario (Tile-covered street) because no real obstacle prevented the AV’s movement. We expect this command to be handy when sensors wrongly recognize imaginary obstacles. Additionally, the “Continue” command can be helpful after resolving road issues, such as removing barriers from the road (3^rd and 4^th scenarios).

Progress Slowly (19) – is a command that guides the AV to continue very slowly and allows immediate stoppage in case of need. RO14 suggested “… progress slowly until the situation becomes clearer …” in the 1^st scenario (Tile-covered street), and RO15 told to “… crawl to see if I can bypass the obstacle without moving from the asphalt …” in the 4^th scenario (A tree blocking the road).

Stop & Wait/Wait (12) – This command entails that the AV should stop and suspend its progress for some time. The amount of time an AV should break its movement can be predefined or controlled manually by the RO. For instance, RO4, RO10, and RO15 suggested stopping and waiting during the resolution of the 4^th scenario (A tree blocking the road). Such action is essential if the AV should wait for assistance from a 3^rd party or for an enthusiastic passenger who volunteered to move the tree branch from the road.

Set speed (12) – The participants wanted to control the AV’s movement speed in several scenarios. RO1 suggested reducing speed when entering a tile-covered street, and RO12 told to “… Snap to the right, go at a constant speed, and don’t perform overtakes …” when the lanes are not visible (5^th scenario).

Stop/Emergency Stop (6) – The “Stop” command should immediately stop an AV in case of need. RO7 shared with us that “… the stop button should be visible. We had an emergency stop button connected to the AV via a separate networking channel …”

Gap Keep (4) – is a command that enables the RO to request from the AV to keep a constant distance (of her choice) from a selected vehicle. RO13 gave an excellent description of the desired interaction as a mitigation for the 8^th scenario (Integration into heavy traffic): “… You press on a ‘Gap Keep’ button, select which vehicle to follow, and set the distance to X cm …”

4.2.4. Other autonomy modes

In this sub-section, we discuss commands that aggregate several commands that change AV’s general behavior. Such commands are essential because they transfer responsibilities from the human RO to the AV and shorten a single intervention session’s length.

Take Manual Control (72) – Based on our study, at this point of technological advancement in the field of AVs, continuous manual control (i.e., tele-driving) will be necessary as a fallback in many road situations. Therefore, an easy transition between higher-level commands and manual control is essential.

Changing operation modes (12) – Nine interviewees mentioned a transition to a different automatic driving mode as a possible mitigation technique for various road scenarios that require remote human intervention: RO3 and RO9 suggested using “Cautious Driving Mode” when entering a tile-covered street (1^st scenario), RO17 mentioned an “Off-road Driving Mode” when bypassing a tree on a none asphalt road (4^th scenario), and RO6 and RO8 mentioned an “Aggressive Driving Mode” to integrate into heavy traffic (8^th scenario). Additional driving modes can and should be defined in the future, but the idea of a higher level of guidance and relaying more responsibilities to autonomy seems valuable.

Applying high-level routines (5) – An additional technique aggregates several high- and low-level commands into a single higher-level command. An excellent example of such a routine was mentioned by RO15 for scenario 10 (Flat tyre): “… when you have a Flat tyre, there should be a routine for this case: activate hazard light, notify the environment [about the problem], and pull over …” Similarly, to driving modes of operation, many more routines can be defined.

4.2.5. RO’s view improvement via sensor control

Zoom/Switch between cameras (16) – In many cases, interviewees wanted to improve their view of the scene. For example, in scenario 4 (A tree blocking the road), RO2 shared that she needs “… to explore the type of the surface before [she does] a maneuver …” and in scenario 6, RO1 said that he “… want[s] to have an option to zoom into the police officer to understand his gestures …”

Switch between views (4) – Several interviewees expressed willingness to change between different views and camera types to increase their remote scene comprehension. Some wanted to see video feeds from left, right, and rear cameras on the big screen. Others – to view the same scene via different sensor types. RO9 shared that she “… should be able to switch between day, night, thermal, and infra-red cameras …” when resolving the bad visibility conditions in scenario 9. While such sensors might be expensive and unavailable in large fleets of robotic taxis, we believe that having the affordance of seeing the same scene through different sensor types is valuable and has the potential to increase the RO’s SA.

4.2.6. Within/between lane placement control

Snap to the right/left (10) – This command directs an AV to move to the utmost right or left lanes in various road situations. The command was primarily mentioned when discussing the 5^th scenario, in which lane marks disappear, and the AV needs to find alternative landmarks on which to base its movements. For instance, RO12 suggested to “… snap to the right, go at a constant speed, [and] don’t perform overtakes …”

Follow orange/white lanes (4) – To address two separate sets of lanes in the 5^th scenario, the participants suggested having a command to distinguish which lane to follow. RO7 described the desired interaction in detail: “… If the AV can distinguish colors, it should tell me that it discovered two sets of lanes (orange and white) and show me two corresponding buttons. I would click on the orange button …. If the AV can’t distinguish colors, the AV should show an AR overlay of lanes. Then I would pick ‘this and this’ lanes as the right ones and press ‘Go’ …”

Deviate right/left [by X meters] (3) – a command that addresses the AV’s placement within its lane. When discussing the 2^nd scenario (Slowly moving vehicle ahead), RO15 noticed that “… the AV is [too] close to the separation lane …” and suggested deviating to the right a bit. Additionally, RO1 mentioned the same command for the 3^rd scenario (An animal blocking the road).

5. Idesign of a teleassistance user interface

Based on the high-level commands, their classification into thematical clusters, and the scenario-commands mapping, we designed an initial user interface to exemplify how these commands can be utilized in an actual tele-assistance station. In Figure 5, we present a scenario in which a static obstacle (a fallen tree) blocks a one-lane road surrounded by non-asphalt roadsides. In this case, the AV cannot break its operational design domain ODD¹⁰ (driving on a non-asphalt road) without explicit permission from a human operator and will require teleoperation assistance from the teleoperation center. In such a scenario, the AV will request assistance. The raised request will be assessed by the teleoperation system to match it with an available teleoperator. The selected RO will receive an alert and see the presented screen (alongside the vehicle’s frontal view, see Figure 1). After receiving the alert and assessing the remote situation, the RO will choose a desired command (e.g., “Bypass from Left”) according to her or his judgment and monitor the maneuver till completion.

Figure 5. Initial tele-assistance user interface design. Suggested high-level commands (on the right side) are according to the elicited results for the “a tree blocking the road” use case, as shown in Table 2.

A tablet-based user interface was chosen because touch interfaces are ubiquitous, intuitive, and quickly adopted as a tool for command issuance. Moreover, other researchers (Kusano et al., 2023) also used a tablet-based input method to design teleoperation stations (Kettwich et al., 2021). We assume that the tablet-based interface will be part of a larger teleoperation station ecosystem (see Figure 1, right), which will also include a large screen (or multiple screens) that presents a live video feed from several (left, front, right, and rear) cameras, a headset, a steering wheel (and pedals) that the RO can use when tele-driving is necessary, and possibly other components such as a map-based screen.

The above UI design incorporates several vital insights and design decisions: (1) The commands (seen as selection buttons on the right side of Figure 5) should be contextual and adaptive, showing only the relevant commands according to the detected intervention reason; (2) The commands are arranged according to the frequency of the commands in the data. For example, the command “Bypass from Right” was mentioned more than any other command when discussing the “Tree blocking the road” scenario; (3) Only the relevant subset of high-level commands is presented to the RO on the main screen. Other commands are still available to employ but are placed in a different UI subsection that is accessible to the RO on demand (in the “More Commands …” tab in the upper right corner of Figure 5); (4) To avoid unnecessary visual distractions, the most commonly used commands, and ones which may apply to all scenarios, are constantly visible in the interface using the white round buttons at the bottom of the screen; (5) As mentioned by the participants, it is imperative to emphasize the intervention reason for the RO (shown here in red on the top part of the screen). This information may be provided by the AV’s perception systems or the fleet management system that the AV is part of; (6) Finally, we chose a “third person perspective” view (Gorisse et al., 2017), which includes a top view of the AV and its surroundings, as this was deemed preferable by most teleoperators in the interviews. The frontal camera view can be seen in the auxiliary screens (see Figure 1, right).

6. Initial evaluation

The results of the elicitation study yielded scenario-command mapping based on inputs from seventeen experienced ROs. Some commands for a scenario were raised by most of the participants, while others were raised only by a few. This evaluation aimed to receive further input regarding the completeness and practicality of the elicited commands and to assess the overall paradigm of command-based AV teleoperation.

6.1. Methodology

We selected three scenarios to evaluate the elicited commands and designed their respective user interface screens (Figures 5 and 6). These scenarios were selected since they are representative of common possible problems and encompass most elicited commands. Each screen included a screenshot from a simulated scenario and the high-level commands that can address the scenario according to our findings (see Table 3). Seven participants from the elicitation part (RO1, RO2, RO6, RO7, RO12, RO13, and RO15) agreed to participate in the evaluation. We note that the purpose of the study was not to evaluate this specific interface design but to take another look at the list of elicited commands and assess the tele-assistance paradigm. The evaluation was conducted remotely via Zoom. For each participant, we first showed each of the designed screens, asking participants to comment on the scenario, given the suggested commands. Next, we asked the following questions:

Figure 6. Initial tele-assistance user interface design for “integration into heavy traffic” (left) and “communication with a law enforcement representative” (right) scenarios.

1. Is guiding a vehicle using discrete high-level commands via a specialized interface, such as this, a realistic approach?

2. Can the following commands be used to resolve the presented remote scenario?

3. Are there any unnecessary commands?

4. Are there any missing commands?

Following the abovementioned interviews, we transcribed the conversations and created a command-feedback mapping for each command included in the design. We then aggregated the documented data into several themes, as described next.

6.2. Results

Most participants believed that operating a vehicle using high-level commands can be very helpful and practical, and the elicited set of commands is sufficient to operate an AV in the presented representative road scenarios. Few participants commented that tele-assistance is desired but context-dependent because (1) Some scenarios cannot be handled using discrete commands, and (2) Guiding AVs using commands gives less feeling of control over the vehicle in comparison to direct driving. Next, we present the most significant themes that emerged from the interviews.

6.2.1. Existing commands related comments

During the evaluation, some of the commands were found not to be well-fit for specific road scenario resolutions:

• Continue – Five participants were reluctant to use the “Continue” command in the “A tree blocking the road” scenario. One participant said, “…I don’t think going over the obstacle is necessarily a good default option. It’s not that the option shouldn’t exist; it’s just that this shouldn’t be the default option …” Additionally, most participants were reluctant to use this command in the “Integration into heavy traffic” scenario because it was too risky.

• Progress Slowly – Much discussion revolved around the “Progress Slowly” command. RO13 implied that “Progress Slowly” is not a clear enough command: “… When we say, ‘Progress slowly’, is the AV going to slide and stop or is it going to move a fixed distance and stop? … You are not trying to drive in real-time with these buttons, so you can always define this command to be ‘move 0.5 meters and stop’ …” RO1 suggested to have two separate commands: “Slide” (can be without stopping) and “Inch Forward.” Another suggestion was to slide as long as the command is pressed (“click & hold” interaction). Finally, RO2 suggested a merge between “Stop” and “Continue” commands and an interchange between them, as it is done with “pause” and “play” commands in video systems.

• U-Turn – three participants noted that performing a U-turn in the “A tree blocking the road” scenario does not make sense because it is a one-way road and because the ground surface (mud) might not allow it. Additionally, four participants were reluctant to use a “U-turn” to resolve the “Integration into heavy traffic” scenario because it didn’t allow them to pass through the junction. Thus, we note that the suggested high-level command language should enable the resolution of complex traffic situations without changing the AV’s original route. If this is not possible, the AV should be capable of rerouting autonomously and in advance.

• Gap Keep – Two participants raised concerns about using this command in the “Integration into heavy traffic” scenario. One participant said that in that exact situation, the “Gap Keep” command did not have any meaning because the vehicle before was already inside the junction. However, this would be useful if several vehicles were ahead of the ego car. Another participant noted that “… It might be a problem to do [Gap Keep] in real-time since [while I was giving the command] the vehicle ahead of me already drove away …” RO2 also mentioned that the leading and the following vehicles might go in different directions “… but in whatever case the AV will just continue on its [own] path …” In other words, while “Gap Keep” was found to be a valid command, it might not be appropriate in the resolution of the upper-mentioned scenario.

6.2.2. Newly suggested commands

During the evaluation, several new commands were suggested; RO13 suggested adding a “Report” command, enabling the RO to note something about a specific intervention: “… [For example] If you run into an incident, there should be a way [not only] to report it to the authorities, but also a way to collect and view the logs … For example, 3-minute footage before and after the incident …” RO2 wanted an option to pass the problem to somebody else in case she cannot resolve the situation: “… [I want to be able to] Raise a ticket … I know I can’t solve this, so I need help …” Another suggestion was to enable textual input in the interface to allow its projection on the AVs body, such as “… I am stopping. You can safely cross the intersection …” which may improve the communication between the AV and the pedestrians around it.

6.2.3. Other significant insights

6.2.3.1. Command design as part of a larger teleoperation ecosystem

When designing the tele-assistance UI, designers should consider this UI part of a larger teleoperation station (see Figure 1). For example, RO6 noted that transitioning to a map view to “Plot Alternative Route” or “Reroute” is problematic because the map hides the video feed in the tele-assistance UI. At the same time, RO12 emphasized the need always to see a map view. One possible solution is to use external screens to show both the map and a continuous video feed from front cameras while dedicating the tele-assistance UI to high-level commands and other perspectives. In addition, several participants mentioned the need for a physical “Emergency Stop” button that should be separate from the tele-assistance UI. RO7 said, “… I'm missing a clear STOP button that stops the AV no matter what …” Another participant emphasized that such a button should allow the RO to react on a visceral level.

6.2.3.2. Discrete commands vs. continuous remote driving

During the evaluation, it became clear that continuous remote driving can be advantageous in some scenarios. For example, five participants claimed that taking manual control in the “Integration into heavy traffic” scenario was better than assisting the vehicle using discrete commands. One participant said that “… In this situation, I don’t want to resolve the situation using semi-autonomous commands. I want to get to a state when I do an action, and the vehicle immediately executes it. I [also] want to be able to change this action. This might not be only a stop because maybe a motorcycle enters the junction, and [then] I want to do an evading maneuver …” At the same time, RO13 said that “… [while] not all scenarios can be handled by discrete commands, [remote driving] should be the last resort …” Therefore, we believe that future teleoperation stations should support both tele-assistance and tele-driving.

6.2.3.3. Improve the effectiveness of the communication commands

Communication between the RO, other systems stakeholders (e.g., passengers), and road users should be as efficient as possible. For instance, RO12 noted, “… If I had to call someone, I wouldn’t do it … I came to drive not to talk over the phone …” and suggested an automatic way to report a hazard [instead of contacting a 3^rd party]. In other words, she suggested speeding up the communication process using “Send Coordinates” and “Hazard Report” commands, which were initially elicited but were not included in the design.

6.2.3.4. Commands for view improvement and sensor control

ROs liked the suggested camera perspective and the options to zoom in/out, see a map screen, and switch between 2D and 3D views. At the same time, participants wanted more freedom and control over the presented scene. One participant articulated it vividly: “… [I don’t have] enough control over the view … One of the most important things is to control what I see … I need specific angles such as 'In Vehicle', 'On the side', 'Rear View', etc. in the fastest possible way. I [also] need all the PTZ (= Pan, Tilt, Zoom) at hand …” Several interesting scene investigation modes were suggested. RO1 suggested having an action called “Point and Look,” which enables zooming into the region that was pointed at. RO7 suggested drawing a rectangle and zooming into the desired section (a magnifying glass interaction). Finally, RO13 suggested utilizing the tablet’s affordances, such as the “pinch” gesture, to explore the remote environment further.

7. Discussion

In this study, we explored a communication language between human operators and autonomous vehicles to enable command-based control of a remote AV. We built upon previous research works that explored disengagement use cases and simulated scenarios to elicit a generic set of high-level commands that an RO can use to resolve various edge-case traffic situations. Our results present a concrete set of commands for ten representative use cases and their general categorization.

When examining the command categorization (Figure 4), we see that the most frequently mentioned commands were related to the communication between the RO and other people. We further divided these into three categories – communication with 3^rd parties (i.e., police, local municipalities, etc.), communication with other road users, and communication with passengers. Communication with 3rd parties (e.g., municipalities, police, etc.) is essential for requesting assistance and reporting various hazards. This emphasizes that teleoperation goes beyond a single teleoperator and requires a more extensive infrastructure that includes a fully connected teleoperation center. The teleoperation interface should consist of appropriate connections with these parties. Communication of AVs with other road users has been investigated mainly in the AutomotiveUI and HCI communities (Lanzer et al., 2020; Nguyen et al., 2019; Reig et al., 2018). Most works have focused on conveying the intentions of the AV to pedestrians and other road users (e.g., when stopping at a pedestrian crossing). The teleoperation interface can leverage these ideas. However, it might be essential to convey to other road users that a remote person is currently controlling and communicating through the AV. Finally, communication with passengers of the AV is also essential. Passengers should be informed of AV’s decisions to feel more confident and trust the AV (Reig et al., 2018). Furthermore, passengers who have a real-world perception of the problem can help with the intervention by providing information to the AV or the RO through an in-vehicle interface. How to design such an interface can be examined in future studies.

The other two most frequently mentioned categories were “Route Control” and “Movement Pace Control.” These categories include various commands to control the movement of the vehicle. One can claim that in some cases, some of these commands, e.g., “Bypass,” can be autonomously performed by Level-4 and Level-5 vehicles. However, it is essential to note that AVs have an operational design domain (ODD) system limiting their driving options. Therefore, situations where the AV must break the ODD must be carefully addressed. For example, in the Tree Blocking the Road scenario (Figure 3, Image 4), the AV will have to overtake it and cross a continuous separation lane. By doing so, the AV breaks the ODD rules. Such an action might have both safety and legal implications, and therefore, an explicit command should be given by the RO. Thus, to enable flexibility of control, the tele-assistance UI should afford such commands in various use cases, as demonstrated by the commands in the above two categories.

In this study, we assumed a one-way communication direction from human to vehicle, primarily involving the issuance of commands through button clicks by the RO. However, collaboration between the RO and AV systems can extend beyond this basic interaction, encompassing more intricate data exchange and a broader spectrum of available interactions. There are various ways in which the communication form between RO and AV can be expanded. First, the AV can provide the RO with critical information from its perceptual engine. For example, the AV can classify objects in the scene (e.g., an animal blocking the road), which might improve RO’s situation awareness and performance if adequately emphasized by the teleoperation interface. In addition, the system can add information gained from external sources to the remote scene. For example, it can alert the RO about an existing vehicle positioned right around a turn (similar to what navigation systems do today). Second, an assisting agent, built as a computational engine that calculates various options according to the capability of the AV, the task at hand, and knowledge of previous ROs’ actions, can provide recommendations to the RO on various tasks. Such an agent can play a pivotal role in assisting the RO by presenting a range of scenario-specific choices for addressing a given situation. As suggested by Trabelsi et al. (Trabelsi et al., 2023), these options could manifest as on-screen overlays for remote drivers, offering visual guidance and suggestions. Another practical example involves proposing multiple alternative routes for circumventing an obstacle obstructing the roadway, as depicted by the green and orange overlays in Figure 5. Thirdly, the semantics of the scene can be analyzed and used to expand current interactions (Majstorovic et al., 2022). For instance, the AV could prompt the human operator to identify or confirm the semantics associated with an obstructing object on the roadway, such as discerning between a flying plastic bag filled with air or a sliding rock. This “semantic tagging” process could benefit immediate decision-making and enhance the knowledge base of other vehicles when they encounter the same location.

Moreover, the selection of objects within the remote environment carries utility beyond human tagging, as it can also serve as a mechanism for issuing commands related to the chosen objects. For example, an RO might select a specific vehicle in the scene and request the AV to follow that vehicle using the “Gap Keep” command. The tele-assistance UI can also offer diverse means for specifying the desired movement trajectory. For instance, the RO could designate a destination to which the AV should navigate using a “Point and Go” command or sketch out a continuous path for the vehicle using the “Plot alternative route” command: these and various other interactions between the RO and the AV warrant further exploration and investigation.

Finally, while we believe tele-assistance has multiple benefits over direct driving, we must acknowledge that discrete commands will not fully substitute direct driving. For example, manually driving the car would still be needed in harsh weather conditions (e.g., heavy rain, as presented in Figure 3, image 9). Thus, hybrid teleoperation stations might be the best option for future teleoperation stations, including direct driving and indirect guidance.

7.1. Limitations and future work

The current work has some limitations. First, we chose ten representative use cases shown to participants to elicit commands. The scenarios were selected from across the use-case categories to be comprehensive according to previous studies. Still, while we noticed that the suggested commands started to repeat themselves, other use cases might elicit other commands. Furthermore, in simulating the use cases, we were limited by the technical possibilities of the simulation software. For example, we could not simulate a deep puddle in the middle of the road or a policeperson who redirects traffic using her hands. Thus, the current list of commands may not be exhaustive.

In some cases, the interviewees related to the specifics of the video image instead of looking at the presented scenario more holistically. For example, RO16 pointed out that the AV was too close to the yellow separation line in the 2^nd scenario (Slowly moving vehicle) and suggested commands that could address this situation. Finally, the fact that the tele-assistance paradigm might be advantageous over tele-driving in certain road scenarios is yet to be proven. In future research, an objective evaluation of tele-assistance measuring RO’s performance, cognitive load, safety, and situation awareness while comparing it to tele-driving is required.

While the presented commands are deeply rooted in real-world scenarios and in the accumulated knowledge of experienced teleoperators, there is still much to be done to translate these commands into a working user interface prototype. In Figures 5 and 6, we presented a suggested design to show how such commands can be used in an interface. Future work would extend and evaluate this interface with simulated scenarios. Additionally, the command language can be further expanded based on the presented categories and beyond. Therefore, the above results can be used both as a starting point for future design explorations and as an extensible framework for expanding a communication language between humans and AVs.

8. Conclusion

In recent years, the teleoperation of AVs has emerged as a practical and effective strategy to expedite the integration of AVs onto public roadways. Tele-driving and tele-assistance represent pivotal paradigms for remote AV operation, with the former presenting notable challenges and complexities. This work focuses on the latter approach, in which a remote human operator guides the AV through high-level commands rather than direct manual control. This approach offers potential benefits such as shorter teleoperation sessions, adaptability to diverse vehicles, improved safety, and reduced cognitive load for operators. This research proposes an initial set of high-level commands derived from simulated scenarios presented to experienced remote operators. These commands can serve as a foundation for establishing a generic communication language between AVs and ROs, as demonstrated by the initial design of a tele-assistance user interface. We believe that the current work is only a first step in the investigation of tele-assistance interfaces and may be helpful for designers, engineers, lawmakers, automakers, and other professionals from industry and academia who wish to design, develop, and test novel methods to create better and safer ways of transportation.

Acknowledgments

This work was supported by the Israeli Innovation Authority, IDIT PhD fellowship, The Israeli Smart Transportation Research Center, and The Israeli Ministry of Aliyah and Integration. We also thank DriveU and Cognata for their collaboration and, specifically, Eli Shapira’s help throughout this work. Finally, we thank Mrs. Lior Kamitchi Rimon and Mr. Ariel Ratner for their assistance.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Felix Tener

Felix Tener is a PhD researcher and a lecturer in the Information Systems department at the University of Haifa. Felix’s research focuses on designing novel interfaces for the teleoperation of autonomous vehicles. Felix received his MS from Georgia Institute of Technology and his BS from the Technion.

Joel Lanir

Joel Lanir is an associate professor at the Information Systems Department at the University of Haifa where he directs the Human-Computer Interaction lab. His research interests lie in the general area of human-computer interaction, and specifically, mobile and context-aware computing, technology for cultural heritage, and information visualization.

Notes

1 https://waymo.com/- Waymo is formally known as the Google self-driving car project.

2 https://zoox.com/- Zoox is a subsidiary of Amazon developing autonomous vehicles that provide Mobility-as-a-Service.

3 Disengagement - a situation when the vehicle returns to manual control or the driver feels the need to take back the wheel from the AV decision system.

4 SAE – Society of Autonomous Engineers (https://www.sae.org/).

5 Tomcar - a type of commercial off-road utility vehicle.

6 https://www.cognata.com/.

7 The “Flat tire” scenario was described verbally since it was very challenging to simulate it.

8 A fleet management center is a center that belongs to the fleet operating company (e.g., Uber) and responsible for fleet maintenance and service providing.

9 https://en.wikipedia.org/wiki/High-definition_map.

10 SAE J3016 defines ODD as “Operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.”.