What rules? Framing the governance of artificial agency

ABSTRACT

Artificial Intelligence (AI) is not new, but recent years have seen a growing concern about the technology’s political, economic and social impact, including debates about its governance. This paper describes how an analysis of the technology’s governance should build on the understanding of AI as the creation of artificial agents, and that the challenge that governance seeks to address is best understood as one of material agency. It describes how relevant rules can be systematically analyzed across different applications by considering the fundamental properties of artificial agents. The paper concludes by describing how the framework can be applied for further governance studies, and as a means to bridge insights across social science and technical perspectives on AI.

KEYWORDS:

• Artificial intelligence
• AI
• governance
• agency
• artificial agents
• algorithms
• ethics

Introduction

While artificial intelligence (AI) is not new, concerns about the technology’s political, economic and social impact have grown in recent years, including debates about its governance. This also overlaps with a broader research agenda on the social impact of algorithmic decision-making that now permeates a range of societal processes; from online search services to high-frequency trading and autonomous vehicles. While the social impact of AI and algorithms has received increasing attention by social science scholars in recent years, less focus has been put on the institutions and processes that govern their development and deployment (Radu, 2021; Ulnicane et al., 2020).

This paper argues that an analysis of the technology’s governance is premised on an understanding of AI as the creation of ‘artificial agents’, and that the challenge that governance seeks to address is best understood as one of material agency. It describes how fundamental properties of artificial agents can be used in a conceptual framework to systematically analyze governance across a vast range of AI applications. As such, it provides a bridge across the literature on AI and the literature on governance, thereby complementing the existing corpus in two important regards.

First, there is a growing literature which provides important insights about the broad range of social challenges that accompany algorithmic systems like AI (e.g. Barocas, Hood, & Ziewitz, 2013; Diakopoulos, 2015; Dickinson et al., 2021; Gillespie, 2014; Just & Latzer, 2017; Tan & Taeihagh, 2020). By connecting these works to the governance of AI the paper elucidates that the governance challenge is less about the system’s absolute capacity, but rather the material agency it exhibits in a given context. The notion of ‘intelligence’ is, in this regard, simply a nebulous success metric for the system’s performance, while the key governance challenge is better described as one of artificial agency.

Second, while important contributions have looked at the governance of algorithms and AI in relation to ethical concerns of accountability and transparency (e.g. Mittelstadt, 2016; Sandvig, Hamilton, Karahalios, & Langbort, 2014), compatibility with existing law (e.g. Leiman, 2020; Pagallo, 2017; Veale, Binns, & Edwards, 2018), and individual applications like autonomous vehicles (Taeihagh & Lim, 2019), the field is seemingly lacking both a lens for the systematic inquiry into AI’s governance, and an understanding of its broader governance system. By using the notion of artificial agents as the common denominator across various applications, the paper draws from the governance literature to propose a conceptual framework that is applicable to all types of artificial agents. In doing so, it can operationalize the governance of AI as intersubjectively recognized rules that define, constrain, and shape expectations about the fundamental properties of an artificial agent.

The paper is organized as follows. The first section provides an operationalization of AI by conceptualizing the task as the creation of artificial agents, which is in line with the dominant approach to the design of AI systems. This concept is then used as the key reference for those artifacts commonly referred to as AI. However, for some sections, it will also be used interchangeably with the term ‘algorithm’ – both because algorithms constitute the agent’s decision-making logic and because popular usage of the term rarely relate to the algorithm as a mathematical construct but rather its implementation in a technical system (Mittelstadt, Allo, Taddeo, Wachter, & Floridi, 2016; Yeung, 2018).

The second section of the paper outlines the notion of artificial agency as the key challenge that governance seeks to address. In drawing from the existing literature on the social ordering capacity of algorithms, it frames the issue as inherently related to the material agency of artificial agents. As such, it connects to a longer tradition within the social sciences that recognize material objects as being explanatory variables of their own (e.g. Latour, 2005; Leander, 2013; Pinch & Bijker, 1984).

The third section demonstrates how insights from the governance literature, combined with an understanding of the fundamental properties of artificial agents, can be used to produce a conceptual framework for AI governance. Using empirical examples of existing and emerging mechanisms, it illustrates how the framework is applicable to all types of AI applications – whether in physical or digital environments.

The paper concludes by mapping how the conceptual framework can both be applied for further studies and for linking future and past research on the governance of AI.

AI as the creation of artificial agents

No precise or universally accepted definition exists for AI (Stone et al., 2016). Instead, the field is broadly defined as ‘devoted to making machines intelligent’ (Nilsson, 2009), suffering from the paradoxical fate that any progress made ‘inevitably gets pulled inside the frontier’ as people become used to new technologies (Stone et al., 2016).

While a moving target in terms of perceptions of the frontier, the general idea that encompasses AI is to create machines that mimic cognitive functions such as learning, the ability to process natural language, to reason, and to perceive the environment. The ‘mimicking’ of intelligence is, in turn, operationalized through algorithms that instruct the behavior of a computer to act ‘intelligently’ (Acemoglu & Restrepo, 2020). The role of algorithms, abstractly defined as sets of defined steps to process data into a desired output (Kitchin, 2017), is consequently central to the concept of AI.

The fact that algorithms are implemented through computer code means that algorithms are in turn dependent on their surrounding architecture, such as programming language or hardware, but also the data upon which they operate. This is particularly true in so-called machine learning, where computers are programmed through the automated detection of meaningful patterns in data by using ‘learning algorithms’ that, in turn, formulate new instructions (Shalev-Shwartz & Ben-David, 2014; Taeihagh, 2021). This approach is itself one of the cognitive functions that AI seeks to mimic, but importantly it also allows for the ability to mimic other functions such as processing natural language or image recognition by providing the learning algorithm with large data sets of examples.

The dominant approach for the creation of AI is one whereby the task is conceptualized as the creation of artificial agents, embodied either as software in a digital environment or as robots in the physical world (Poole & Mackworth, 2017; Russell & Norvig, 2009). Defined as ‘anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators’(Russell & Norvig, 2009, p. 34), artificial agents are in this view central to AI.

Intelligence is in turn operationalized as the creation of rational agents, which means that the agent is ‘doing the right thing’ given a particular performance measure (ibid). The property of learning is an important feature for rational agents since it allows the agent to rely on its percepts rather than initial configurations; allowing the agent to act with a greater degree of autonomy and to solve more complex tasks. The idea of ‘intelligence’ is in this regard just a nebulous success metric, while what is important is the ability to learn (Tutt, 2016) and to optimize a given performance measure.

Fundamental properties of an artificial agent

The seminal textbook on AI by Russell and Norvig (2009, p. 40) describes the most basic step of designing an artificial agent as specifying the ‘task environment’, understood as the ‘problems’ to which the artificial agent constitute a ‘solution’. Derived from the basic definition of an agent this can be thought of as a structured approach for outlining the most critical considerations for the agent’s design. Summarized with the acronym ‘PEAS’, the task environment specification consists of:

• Performance measure: What is good behavior by the agent?

• Environment: What environment does the agent interact with?

• Actuators: How does the agent affect its environment?

• Sensors: How does the agent get information from its environment?

The fact that the task environment is formulated at an abstract level means that it is applicable to all artificial agents, irrespective of the complexity of the task or whether the agent is embodied as a software acting in a virtual environment, or a robot operating in the physical world. In addition, Russell & Norvig also describe how various properties of the environment function as important determinants of the artificial agent’s design. For example, if the environment is fully or partially observable (e.g. a chessboard is fully observable while road traffic is not), or deterministic or stochastic (whether the next state of the environment is completely determined by the action of the agent or by a probability).

The task environment is significant because of the lens it provides to make visible considerations for the agent’s design and deployment. As such, the task environment can also be used in a descriptive account of any artificial agent, as illustrated by the below examples.¹

Table 1. Example of three agents according to their task environment

Agent type	Performance measure	Environment	Actuators	Sensors
Autonomous vehicle	Safe, fast, legal, comfortable trip	Roads, other traffic, pedestrians, passengers	Steering, accelerator, brake, signal, horn, display	Camera, lidar, speedometer, GPS, engine sensors
Image classifier	Correct image categorization	Database of images	Display of scene categorization	Color pixel arrays
Part-picking robot	Percentage of parts in correct bins	Conveyor belt with parts, bins	Jointed arm and hand	Camera, joint angle sensors

However, for the purpose of studying the governance of artificial agents, this lens provided by the PEAS specification can also be viewed as encompassing the fundamental properties of an AI system. Perhaps counterintuitively this includes the agent’s environment. Both because it operates as a defining characteristic of the agent, but importantly because the system’s behaviors and agency are intrinsically linked to a given environment. In this sense, they constitute the four properties that are applicable to any artificial agent, where variations along one or all of the four properties changes the nature of the artificial agent and the agency it asserts.

It should be noted that, in practice, an artificial agent may itself be a multi-agent system, whereby sub-agents perform functions in accordance with their local goals. For example, an autonomous vehicle is itself conceptualized as an agent, but as a technological system it may incorporate lower-level agents like an image classifier to identify and separate between different objects in the environment (e.g. between a pedestrian and a plastic bag). Consequently, successful behavior by higher-level agents, such as the driver agent in charge of maneuvering the vehicle, is dependent on the performance of lower-level agents in the system. Thus, the performance measures described above, like ‘safe’ and ‘comfortable trip’, should be seen as macro-level goals that are supported by sub-level goals such as ‘smooth braking’. While this introduces varying degrees of complexity to the analysis, it does not change the ability to describe the individual (sub) agents using the four properties.

The Challenge of AI: Artificial Agency

AI is, by a technical definition, characterized as artificial agents capable of acting autonomously within a given environment, but the notion that artifacts can be described as actors and discussions of material agency has a longer tradition within the social sciences (e.g. Latour, 2005; Leander, 2013; Pinch & Bijker, 1984). At its core is a descriptive account of material objects to be considered as explanatory variables of their own, and to be considered as actors in a given analysis. As described by Lehrer et al (2018), ‘[m]aterial agency describes a technology’s capacity to act on its own, apart from human intervention, while human agency refers to humans’ capacity to form and realize their goals’. It is as such not an anthropomorphic account, but a recognition that material objects can be said to act in the sense of affecting social ordering.

What artificial agents do

Artificial agents are deployed in a variety of contexts and environments that include both the physical and the digital world. In these, algorithms act in the most basic sense by ‘selecting information elements and assigning relevance to them’ (Latzer, Hollnbuchner, Just, & Saurwein, 2016). What an artificial agent actually ‘does’ in the sense of having agency is thus highly contextual. In the context of, for example, online services, algorithms rank information, suggest friends and provide offers of products and services. In the physical world, algorithms operate vehicles, regulate homes, surveil, and much more. As such, algorithms are deployed as economic and political instruments for different activities across a range of domains, what Winner would describe as political artifacts in that their design establishes power and authority in a particular setting by being ‘forms of order’ (Winner, 1980).

However, the fact that artificial agents are deployed in an instrumental capacity towards certain objectives does not mean that they are merely intervening variables for socioeconomic outcomes. Instead, artificial agents often hold an explanatory role in and of themselves. The most apparent example stems from the fact that algorithms constitute a form of delegated decision-making that is capable of systematically evaluating data at scale. For example, consider an algorithm to determine insurance premiums based on high-dimensional data with hundreds or thousands of features to be considered in a decision. At face-value such a decision is no more than a substitute for work that has previously been undertaken by a human, but is arguably adhering to a different decision-making logic capable of considering much greater inputs (Mittelstadt et al., 2016). Similarly, autonomous vehicles perform a task that may equally be conducted by a human, but relies on different decision-making metrics from calculations of input from cameras, GPS and other sensors.

Algorithms operate based on limited and pre-defined representations of the world, understood as the data upon which they make decisions. As Gillespie (2014, p. 169) expresses it, ‘[a]lgorithms are inert, meaningless machines until paired with databases on which to function’. It is in pairing algorithms with data that they become meaningful, but they also come to generate meaning depending on the underlying data. For example, datasets that incorporate variables that have structural dependencies may result in discriminatory decisions or actions by AI (Zliobaite, 2015).

However, the ways in which algorithms generate meaning goes beyond intentional or unintentional design. Algorithmic decision-making may also re-ontologize the world ‘by understanding and conceptualizing it in new, unexpected ways, and triggering and motivating actions based on the insights it generates’ (Mittelstadt et al., 2016). The most concrete example is the ways in which artificial agents construct the available action space of online environments, such as search engines that make available links to other websites through an algorithmic ranking. Algorithms are, in this regard, part of a process of ‘reality construction’ by including or omitting specific information that, in effect, governs behavior and actions (Just & Latzer, 2017). Similarly, what a search engine lists in search results, and the way in which it ranks those results by dictating systematic ‘prominence for some sites’ and ‘invisibility for others’ (Introna & Nissenbaum, 2000) is arguably a political issue of granting selective visibility. In some cases, this challenge can be further exacerbated by ‘performative prediction’ in which the prediction itself influences the outcome that is predicted (e.g. if the prediction of stock prices influences trading activity, and in consequence prices) (Perdomo, Zrnic, Mendler-Dünner, & Hardt, 2020).

Whereas the case of search engines provides an illustration of how artificial agents may re-ontologize the world, their impact extends beyond modifications to the digital world. Predicative analytics applied to policing offer one example of how algorithms may influence perceptions (and actions) towards whole social groups by literally designating parts of the physical world as more or less dangerous through classification of risk (Brayne, 2017) . Artificial agents are, in this sense, actors that conceptualize the world in a manner that enables, produces or encourages new modes of social and political organization.

The role that algorithms play in producing outcomes for interpretation also reveals their role in producing knowledge. Beyond the inherent limitations of acting on a limited representation of the world is the fact that algorithmic decisions rely on correlations within datasets, which may result in interpretations of causation that shape an understanding of a topic. Mittelstadt et al. (2016) describe this as one of the ethical challenges with algorithmic decisions as it may result in ‘inconclusive evidence leading to unjustified actions’. Underpinning this argument is a perception that correlations based on so-called ‘big data’ are increasingly seen as sufficient to guide action, without establishing causality. In terms of agency, algorithms can be seen as actors in the production of knowledge by shaping what is perceived as sufficient evidence to render ‘actionable insights’ – potentially with ethical implications (ibid).

The fact that artificial agents, through the computational generation of knowledge, can constrain, alter and nudge behavior towards a specific goal has also been conceptualized as ‘algorithmic regulation’ (Yeung, 2018), ‘governance by algorithms’ (Barocas et al., 2013; Just & Latzer, 2017), ‘artifacts of governance’ (Musiani, 2013), and ‘algorithmic governmentality’ (Introna, 2016; Rodrigues, 2016).

One implication of this ability to shape behavior is that, for example, norms are not only embodied in artificial agents, but algorithmic decisions may also impose normative rules. Schulz and Dankert (2016) exemplify this with the case of autonomous vehicles that will ‘self-execute’ rules that may restrict or substitute human behavior. Given that the algorithm may need to embody rules of what to do in ethical dilemmas, which can range from life-and-death situations to less severe cases, this means that decisions that law had previously left to a matter of justification become systematically encoded through algorithmic rules (Schulz & Dankert, 2016).

How they do it

While the notion that algorithms ‘do’ things in the world may be sufficient to characterize artificial agents as artifacts that have agency, it is worth noting specific aspects of artificial agents that emphasize their agency as related to their autonomy. The most apparent example is the approach of machine learning, whereby the algorithmic decision-rules are defined and modified autonomously through inference from data (Matthias, 2004). From a technical perspective this is a feature of how AI is operationalized, but it entails a delegation of formulating the decision-making rules to the agent itself. In terms of agency this can be seen as having two major dimensions.

First, actions by an artificial agent may be difficult to predict, or to explain afterwards. The case of Google Photos’ image classifier that labelled some black people as ‘gorillas’ is one such example where the agent had generated its own classification rules resulting in unwanted behavior. The problem proved so difficult to resolve that Google reportedly removed the ‘gorilla’ category all together (Simonite, 2018).

The second issue encompass the challenges of transparency and interpretability of algorithmic decisions. In some instances, the challenge of transparency results from corporate secrecy and intellectual property rights (Burrell, 2016), but the ability to fully comprehend the intimate workings and rules of algorithms can also be limited due to the scope, length or level of expertise needed to interpret the code (Kitchin, 2017; Taeihagh, 2021). In other instances, and in particular for techniques like modeling through neural networks, algorithms are often hard to understand even for their creators (Burrell, 2016). The fact that some algorithms modify their decision-rules during operation render the rationale of the algorithm obscure, resulting in some machine-learning algorithms often being referred to as ‘black boxes’ (Mittelstadt et al., 2016, p. 6). Opacity is, in this regard, a product of the high dimensionality of data, complex code and changeable decision-making logic, with the result that not even the human trainer may be able to ‘provide an algorithmic representation’ (Matthias, 2004:179; Taeihagh, 2020).

Taken together, the fact that agents may act autonomously in unpredictable and highly opaque manners imply a further move towards artificial agents holding an explanatory role in and of themselves. Combined with the social ordering capacity of algorithms, one can therefore think of artificial agency as the ability of artificial agents to be delegated decision-making, to re-ontologize the world, and to regulate behavior in a manner that is both unpredictable and opaque. This agency is the phenomenon that governance seeks to address.

The governance of AI – a purposive order that steers its fundamental properties

Governance can be seen as that part of human activity that strives towards ‘creating conditions for ordered rule and collective action’ Stoker (1998), on an abstract level, it is about the processes of social coordination and organization (Bevir, 2012). The term can be used to denote all patterns of rule, whether at the local or international level, and goes beyond activities of the state (Bevir, 2009). Rather, it is a horizontal or vertical extension of the state, and a purposive order aimed at ‘steering’ a community towards a particular goal or goals.

This order is characterized by a system of rule(s) that may be formal, such as laws or regulation, but also encompass informal rules like norms (Biersteker, 2009).They may be restrictive by constraining particular activities or actions, or prescriptive and normative in defining what should be done. In line with Finnemore and Sikkink (1998) discussion of norms, they may also be ‘constitutive’ in the sense that they create new actors, interests, identities, or categories of actors. Thus, as described by Biersteker (2009), governance encompasses a system of rules that ‘defines, constrains and shapes actors’ expectations’.

Governance is also a relational concept that describes a constituency’s behavior with regard to the system of rules, which amounts to a social relationship between the governed and a governing authority (ibid). The important implication is that the authorship of rules in a governance system need not be formulated by the state but may equally stem from private actors, provided they are recognized by the governed constituency as legitimate and ‘ought to be obeyed’ (Hurd, 1999).

Importantly, governance relates to an observable phenomenon of rules that govern behavior in a given issue domain (Hufty, 2011), and the study of governance is therefore neither normative nor prescriptive but concerned with the purposive order as a social fact. For example, in the study of transnational organized crime, governance may even be exercised by an ‘illicit authority’ that enjoys a legitimate social recognition to the extent that they step into a power vacuum left by a weak state (Williams, 2002).

As outlined in the previous sections, the domain of AI is in practice about the creation of artificial agents, for which the governance challenge is the material agency they possess. While this agency is contextual, it is also inherently the result of the agent’s design. Thus, to govern this agency is consequently about governing the technology’s deployment and design, for which important considerations can be viewed through the lens of an agent’s fundamental properties. Through examples of public and private governance mechanisms with a bearing on these properties, the following section illustrates a conceptual framework applicable to all artificial agents in both physical and digital environments. In doing so, and paraphrasing Biersteker’s (2009) definition of global governance, it operationalizes the governance of AI as those intersubjectively recognized rules that define, constrain and shape expectations about the fundamental properties of an artificial agent.

Rules that define, constrain and shape the performance measure

The performance measure of an artificial agent denotes what outcome constitutes success. As such, it is intrinsically linked to the system’s decision-making logic, which translates higher level performance measures to the agent’s actual behavior. Thus, it is possible to have two agents operating in the same type of environment, able to take the same type of actions, and receiving the same information from the environment, but relying on different decision-making logics. For example, one autonomous vehicle could rely on decision-making criteria and constraints that seeks to optimize a performance measure related to safety, while another optimizes one related to speed.

Performance measures are implemented by whomever designs or deploys the agent, but this does not happen in a social vacuum. Governance can, in this view, be illustrated by rules that influence what constitutes a good, or perhaps even legal, performance measure in a given environment. Just like any domain of social activity there is a presence of formal (e.g. laws and regulation) and informal rules (e.g. norms) that denote, and shape expectations of, good or acceptable behavior in a given context.

A clear example of a governance rule with a direct impact on the performance measure can be found in efforts to establish a norm against the use of ‘killer robots’, i.e. fully autonomous lethal weapons systems that have harm to humans (within certain parameters) among its explicit success criteria. Such a norm is currently under discussion by the United Nation’s Group of Governmental Experts on Lethal Autonomous Weapons Systems, which is tasked with exploring and agreeing on recommendations for the use of autonomous weapons systems under international law. However, the norm does not need to be codified by a public institution to have an effect, but could also result in voluntary self-governance by private companies. This is exemplified by the thousands of AI researchers, and hundreds of AI-related organizations, that have signed a pledge committing to ‘neither participate in nor support the development, manufacture, trade, or use of lethal autonomous weapons’ (Jenkins, 2018).

An illustration of formal rules that directly constrain, but also shape expectations about, performance measures are regulations related to the ‘explainability’ of algorithmic decisions. Such requirements on the system’s decision-making logic act as a constraint by conditioning certain performance measures on the ability to explain how a prediction was made. These types of regulations are not new but can be traced back to the 1970s and the US’ Equal Credit Opportunity Act and the Fair Credit Reporting Act, under which customers are guaranteed notifications of reasons for adverse actions, including those based on automated scoring systems. Similarly, the 1995 EU Data Protection Directive provided individuals with a ‘right of access’ to demand ‘knowledge of the logic involved’ in automated decision-making, e.g. about creditworthiness (Wachter, Mittelstadt, & Floridi, 2017). This provision was replaced by the 2016 EU General Data Protection Regulation (GDPR) and by what has become popularly known as ‘the right to explanation’ (Goodman & Flaxman, 2017; Selbst & Powles, 2017).

Both examples above describe governance efforts that shape the decision-making logic at an abstract level. As long as the overall logic adhere to the constraints, of being explainable and of not supporting a particular performance measure, there is nothing that prescribes what the individual decision-rules should look like. Recommendations from the German Ethics Commission on Automated Driving, addressing the behavior of autonomous vehicles in unavoidable accident scenarios, provide an example that goes to such level of detail. The recommendation states that any distinction on the basis of personal features (like gender and age) should be prohibited, and that victims should not be offset against one another (FMTDI, 2017). Hence, it is much more detailed since it targets individual decision-rules, imposing constraints at the level of formulating computer code.

While governance rules may define the ‘problem’ to be considered in the design of an artificial agent, there are also governance mechanisms that support solutions. For example, the Institute of Electrical and Electronics Engineers’ (IEEE) P7001 standard for ‘Transparency of Autonomous Systems’, which is currently under development, aims to ‘describe measurable, testable levels of transparency, so that autonomous systems can be objectively assessed and levels of compliance determined’ (IEEE-P7001, n.d.). Such a standard could become applicable for compliance with regulations like the GDPR, and thus constitutes a solution to a particular problem. However, it is also a mechanism with wider applications as stipulated by the purpose description, for example to build public confidence and to support accident investigations. The common denominator is that it provides a standard for those artificial agents with a performance measures that include explainability or transparency as a metric of success.

Rules that define, constrain and shape the environment

The environment concerns the context in which a particular agent is operating. An autonomous car operates in an environment of roads, pedestrians, traffic signs; software operates in a computational environment that may consist of a database of images or websites; and so on.

A clear example of rules that govern the environment can be found in the case of autonomous vehicles. As an application of AI that is still under development, autonomous vehicles today require permissions for deployment in specific test zones, whereby the agent is limited to only operate within a designated environment. For example, Volvo’s Drive Me project for testing autonomous vehicles in Gothenburg is limited to a total of 50 km of road in and around the city (Lindholmen Science Park, 2013). From the perspective of governance, these rules are specifically focused on defining the environment in which an agent operates, but they do not address other properties of the agent, such as the performance measure or what actions it may take.

A similar example for a virtual agent can be found for the use of ‘bots’ in online platforms. Online poker websites, for instance, tend to prohibit the deployment of artificial agents. Websites like Full Tilt explicitly state in their terms of service that ‘[a]ll actions taken in relation to the Service by a User must be executed personally by players through the user interface accessible by use of the Software, and without the assistance of any form of artificial intelligence’ (Full Tilt, 2019). Similar to the case of autonomous vehicles, this restriction on the use of bots illustrates rules that govern artificial agents by constraining the environment for which they can be deployed.

Whereas governance in regard to an agent’s environment can be described in terms of constraints or prohibitions as illustrated above, certain governance mechanisms also support solutions to problems related to a given environment. For example, if an agent is to learn its behavior from training data for a given environment, its success is dependent on the training data providing a good reflection of the actual environment. Such challenges can have wide-ranging implications, not least ethical, as illustrated by the concerns of algorithmic decisions reflecting biases of the training data (Diakopoulos, 2015; Lim & Taeihagh, 2019).

An example of efforts to provide a solution to such challenges is the non-profit organization Algorithmic Justice League (AJL), which offers services that help create ‘inclusive training sets’ for machine learning. The founder of the organization, Joy Buolamwini, describes this challenge through her own experience of not having her face recognized by facial recognition software that had been trained on data of predominantly white faces (AJL, n.d.). The creation of inclusive training sets is in this view understood as providing the artificial agent with training in an environment that provides a better reflection of the real world. These solutions form part of a broader general consensus in the AI policy discourse on the importance of inclusivity and diversity in the design of AI systems to ensure that its benefits accrue to all segments of society (Ulnicane et al. 2020; Radu, 2021). This mechanism therefore does not specify what the performance measure should be, but defines what the environment should consist of.

Rules that define, constrain and shape the agent’s actions

Agents are driven by whatever performance measure has been stipulated as success, but may take different actions in pursuit of a particular objective. For example, two autonomous vehicles may operate in relation to the same performance measure (e.g. safety), in the same environment (e.g. the same city), and with the same information (e.g. the same type of cameras, the same image classifier, etc.), but differ with regard to what actions they can take (e.g. one requires that a human control the accelerator).

An illustration of such governance mechanisms is the taxonomy of different levels of automation for autonomous vehicles developed by the Society of Automotive Engineers (SAE). Specified in the international standard SAE J3016, the taxonomy describes six levels of automation, ranging from manual vehicles to fully automated ones, and describes what type of actions has been delegated to the vehicle at each level. For example, for vehicles classified as Level 1, ‘some driving assistance features may be included that can assist the human driver with either steering or braking/accelerating’, whereas at Level 2 the vehicle has ‘combined automated functions, like speed control and steering simultaneously’ (Campbell et al., 2018). The standard has already been adopted by the US Department of Transportation, specifying in its policy document for automated vehicles that manufacturers and other entities have a ‘responsibility to determine their system’s AV level in conformity with SAE International’s published definitions’(NHTSA, 2016).

Similar examples of rules specifically aimed at the actions of an agent are also available for virtual agents, such as the case of Twitter and their policies governing the use of bots on its platform. Unlike the example of poker bots described above, Twitter does not ban bots from their platform, and there are numerous examples of popular bots that are seen to add value to the service (e.g. bots that automatically post computer-generated poetry). However, and as shown by the use of bots in relation to the Brexit vote and the US 2016 presidential election (Lomas, 2017), bots can also be used for malicious activities and to spread misinformation. To this end, Twitter’s policies specifically prohibit the use of its service for activities like spam or abusive behavior (what could be characterized as rules related to a performance measure), but also include very specific rules as to what actions a bot may take. For example, ‘sending automated replies to Tweets based on keyword searches alone is not permitted’, and nor is ‘liking Tweets in an automated manner’ (Twitter, 2017).

Again, while the two examples above can be described as restrictive rules that contribute towards shaping the ‘problem’ in a given task environment (by restricting what actions are allowed), there are also governance efforts to promote solutions. For example, a standard currently under development by the IEEE, called ‘Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems’ (P7009), aims to develop ‘procedures for measuring, testing, and certifying a system’s ability to fail safely’ (IEEE-P7009, n.d.). As such, this is as an effort targeted at the actions of the agent, or specifically to stop certain actions.

Rules that define, constrain and shape the agent’s percepts

An artificial agent relies on percepts (information) from the environment to take a particular course of action. This can be described as a distinct governance focus by using the case of two agents: they both have the same performance measure, operate in the same environment, and can take the same actions, but they receive, or have access to, different information from the environment.

The typical illustration of formal rules that constrain the agent’s percepts is privacy regulations such as those contained in the GDPR. From the view of the agent, these regulations imply that some information from the environment, e.g. about an individual’s name or address, is not accessible to the agent when taking actions towards a particular goal. For example, an agent tasked with personalizing ads would benefit from as much information as possible about the targeted individual, but is constrained by the fact that regulation prohibits the collection of certain types of data.

A related example is the technical specification ‘Do Not Track’ (DNT) developed by the World Wide Web Consortium (W3C). DNT works through settings in the web browser where a user can indicate their preference for being tracked online – a practice of data collection that is core to the business of personalized advertisement. It entails adding a DNT header to all HTTP requests, where the header value can either start with 1, meaning ‘Do Not Track’, or 0, signifying ‘this user has agreed to tracking for the purposes explained’ (O’Neil, 2018). From a governance perspective, while not widely deployed, it is an attempt to codify and implement a norm that constrains what information can be legitimately retrieved and processed by the agent for the personalization of ads.

While the examples above describe restrictive rules that constrain what information the agent may perceive there are also prescriptive mechanisms that provides solutions. One such mechanism is standards for machine-readable privacy statements, such as the (now obsolete) P3P specification by W3C (W3C, 2018), or a standard currently under development by the IEEE (IEEE-P7012, n.d.). The idea behind these specifications is to make privacy statements available in a format that is readable by machines, thereby allowing, for example, a user’s browser (or other agent, such as a personal digital assistant) to automatically retrieve and interpret a privacy statement on the user’s behalf. If the browser encounters a website that does not support the standard, or has a policy that does not match the user’s preferences, it can alert the user or take other action (Langheinrich, 2002). The key functionality of the standard is thus to allow the agent to retrieve certain information about the environment that it could not retrieve otherwise. This information may be critical for the agent’s ability to achieve certain performance measures (e.g. protecting users’ privacy).

Implications

Before a further elaboration on the implications of the conceptual framework, it is important to note that the above examples are only productive as means of illustrating mechanisms that could be considered governance of artificial agents. To determine the extent to which the identified rules actually govern is an endeavor that demands a closer and more systematic look into whether the rules are adhered to, i.e. if they are in fact authoritative. This include differences of enforceability, such as adding the use of CAPTCHA to enforce an existing rule against automated ‘Likes’ by a social media bot. The examples outlined in previous sections should therefore be considered aspirational governance mechanisms to illustrate how we can understand what type of rules have bearing on governing AI.

However, the examples also show the productive capacity of conceptualizing the governance of AI through the lens of its fundamental properties. As such, and given the high level of abstraction, it constitutes a general framework that is applicable to all agents, allowing for a structured analysis of various governance issues. This includes discussions about governance rules applicable to all artificial agents, but also as a framework to study the governance of specific applications. For example, a detailed analysis of the governance of autonomous vehicles in country X may use the framework to identify the authoritative rules that operate by defining the car’s performance measure, the environments in which it may act, what actions it may take, and what information from the environment it may be required to perceive (e.g. GPS locations of other vehicles). Similarly, governance discussions concerning algorithms in charge of curating information, such as online search or newsfeeds, could equally adopt the framework to systematically analyze existing mechanisms, and identify gaps, by conceptualizing the system as an agent. 

The fact that applications operating as an artificial agent may themselves be a multi-agent system does not undermine the framework’s utility in exploring the governance system involved. Rather, the framework can support the investigation by helping to identify the role of sub-agents (and by extension the governance rules affecting those sub-agents) for the overall system’s performance. For instance, the behavior of a higher-level driving agent in charge of maneuvering an autonomous vehicle may be dependent on the output from a sub-agent tasked with classifying images from a camera. If the sub-agent misclassifies an object in the environment this can affect the behavior of the driving agent since its decision-rules operate on the outputs of the sub-agent. In consequence, governing the safe behavior of autonomous vehicles may require rules, not only for the driving agent, but also the sub-agents involved (e.g. specifying the minimum level of performance for object classifications)

Using the framework also exposes the ‘governors’ of AI by identifying those actors or communities that function as authorities in formulating the related rules and therefore, contribute to future studies on the roles and interests of key actors in the governance of AI (Ulnicane et al., 2020; Radu, 2021). Depending on what level of abstraction the framework is applied, this could be said to encompass any actor performing an authoritative function in formulating rules for the design and deployment of AI. For example, studies at the level of code implementation may find that developers’ communities like Stack Overflow, or software libraries like TensorFlow, perform an authoritative role in shaping standardized solutions with a bearing on the artificial agent’s final behavior.² Similarly, it may also be used to structure analyses at more abstract levels of public discourse about the norms and regulations to govern AI. This also has implications from a practical policy perspective since it can inform issues of responsibility and the need for new accountability mechanisms.

Finally, one of the core strengths of the conceptual framework is that it is applicable to all agents, irrespective of their perceived ‘intelligence’. As such, it is a framework that encompasses future developments in the field, as well as allowing for insights from previous research of agent applications that may not have been framed as AI.

Conclusion

This paper has sought to structure the study of AI governance by introducing a conceptual framework through which the relevant mechanisms can be explored. It has argued that the phenomenon that governance seeks to address is one of artificial agency, understood as the ability of artificial agents’ to be delegated decision-making, to re-ontologize the world, and to regulate behavior in a manner that is both unpredictable and opaque. By drawing on the key characteristic of AI, understood as the overall system’s ability to act as an artificial agent within a specific environment, it has introduced the fundamental properties of an artificial agent as the lens for systematically studying efforts to govern this agency. As such, the paper has drawn from, and connects to, a broader literature on governance by operationalizing the governance of AI as intersubjectively recognized rules that define, constrain and shape expectations about the fundamental properties of an artificial agent. Importantly, the paper opens the door for future research and empirical studies on the governance of artificial agents in general, as well as governance in regard to specific applications.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Carl Gahnberg

Carl Gahnberg is a PhD Candidate at the Department of Political Science and International Relations at the Graduate Institute of International and Development Studies (IHEID) in Geneva. His research is focused on the governance of artificial intelligence, and its intersection with broader technology policy issues.

Notes

¹ Table 1 is adapted from Russell & Norvig (2009:40-42).

² For an illustration of how developer communities may perform an authoritative role see e.g. Fischer et al. (2017) study of copy-paste practices from Stack Overflow, and research by Xiao, Li, Zhang, and Xu (2018) on important vulnerabilities in popular deep learning frameworks including Caffe, TensorFlow, and Torch.