397
Views
0
CrossRef citations to date
0
Altmetric
Research Article

Valuing the manual: the demarcation of embodied practices within algorithmic decision-making processes

Valorando el manual: la demarcación de las prácticas incorporadas dentro de los procesos algorítmicos de toma de decisiones

La valorisation des tâches manuelles : la démarcation des pratiques de la corporéité dans les processus de prise de décision algorithmique

, , , &
Received 21 Nov 2023, Accepted 15 Mar 2024, Published online: 02 May 2024

ABSTRACT

This article takes the notion of the manual as a focal point for understanding the demarcation and evaluation of embodied practices within algorithmic decision-making processes. Focusing on the use of the term manual reveals the limits of automation and the value attached to human intervention. Drawing on extensive qualitative interview data, it is concerned with how those within the UK housing sector approach questions of embodiment and materiality through the lens of the manual. We show how this notion is used to separate human input from algorithmic automation. The article explores how value is attached to the manual and how the term is involved in valuing human interventions. It begins by examining how the manual demarcates spaces and practices outside or alongside the algorithmic. It then focuses upon three manifestations the manual takes within these articulations. First by looking at the manual as the completion of a form, then the manual as a check and finally by looking at the manual as approval. The article develops a central argument concerning processes of valuing and evaluating the manual, that are tied up with these visions and demarcations.

Résumé

Cet article se concentre sur le concept de « manuel » pour comprendre la démarcation et l’évaluation des pratiques de corporéité dans les processus de prises de décision algorithmiques. Une focalisation sur l’utilisation du terme « manuel » révèle les limites de l’automation et la valeur qu’on attache à l’intervention humaine. S’appuyant sur des données issues de nombreux entretiens qualitatifs, il s’intéresse à la manière dont le secteur du logement aborde les problèmes de corporéité et de matérialité à travers le prisme de l’aspect « manuel ». On y montre comment ce concept est utilisé pour séparer la contribution humaine de l’automatisation algorithmique. Il analyse comment on attache une valeur aux tâches manuelles et comment on engage leur concept pour évaluer les interventions humaines. Il commence en examinant la manière dont l’intervention manuelle délimite les espaces et les pratiques en dehors ou en parallèle des systèmes algorithmiques. Il se concentre ensuite sur trois expressions du concept de manuel au sein de ces trois articulations. D’abord, on observe un acte manuel tel que remplir un formulaire, puis celui de la vérification, et enfin on l’examine en tant qu’approbation. Cet article développe une thèse centrale qui concerne les processus de valorisation et d’évaluation des actions manuelles, qui sont liés à ces visions et ces démarcations.

Resumen

Este artículo toma la noción de manual como punto focal para comprender la demarcación y evaluación de las prácticas incorporadas dentro de los procesos algorítmicos de toma de decisiones. Centrarse en el uso del término manual revela los límites de la automatización y el valor que se atribuye a la intervención humana. Basándose en extensos datos cualitativos de entrevistas, el artículo investiga cómo quienes trabajan en el sector inmobiliario del Reino Unido abordan las cuestiones de encarnación y materialidad a través de la lente manual. Mostramos cómo se utiliza esta noción para separar la aportación humana de la automatización algorítmica. El artículo explora cómo se valora el manual y cómo el término interviene en la valoración de las intervenciones humanas. Comienza examinando cómo el manual delimita espacios y prácticas fuera o al margen de lo algorítmico. Luego se centra en tres manifestaciones que adopta el manual dentro de estas articulaciones. Primero mirando el manual como el cumplimiento de un formulario, luego el manual como una verificación y finalmente mirando el manual como una aprobación. El artículo desarrolla un argumento central sobre los procesos de valoración y evaluación del manual, que están ligados a estas visiones y demarcaciones.

Introduction

Given the prevailing visions of immateriality, it may seem a strange moment to turn back to the seemingly outdated concept of the manual. It is a term associated with a more analogue age. Yet the argument we make in this article is that notions of the manual still circulate and have come to perform a particular and defining role in the reconfiguration of automated decision-making processes. Indeed, rather than being rendered irrelevant by new and advancing ‘technoscapes’ (Appadurai, Citation1996, p. 34) we suggest that the manual is a term that has found a new purpose with the expansion of automation and algorithmic systems. The manual – as in working with the hands, rather than a book of instructions - - performs a liminal role by demarcating the edges and boundaries within these developments. Algorithmic systems have, Pasquinelli (Citation2023, p. 1) observes, ‘changed the perception of manual skills’. The notion of the manual has come to be deployed to make sense of the transformations and rethinkings of ‘tactility’ in the context of ‘ubiquitous computing’ (see Ladewig & Schmidgen, Citation2022, pp. 5–6). It is a notion that is used to rationalize and navigate the shifting ‘boundary conditions’ of algorithmic systems and their ‘leaky distinctions’ and ‘boundary constraints’ (Haraway, Citation1991, p. 150). The manual, as we will show, provides a means for handling and ordering these shifting boundary conditions.

So, far from being rendered redundant by seemingly immaterial information systems, the notion of the manual has found new relevance and new applications with the roll-out of automation and algorithms. As this article will demonstrate, the notion of the manual is one means by which embodiment is demarcated and evaluated within the unfolding and mutating processes and practices associated with automation. With the establishment of platforms, data, streaming, algorithms and the like within social ordering, many practices are now understood to be defined by their immateriality, thus lifting them out of their context. We think of them as being conducted online, in the cloud, virtually, and so on. Data-led systems in which algorithms are incorporated into decision-making processes are particularly immaterial within these ‘social imaginaries’ (Taylor, Citation2004). There is now, Bucher (Citation2017) has pointed out, an established ‘algorithmic imaginary’ whilst these systems remain deeply material in their form and outputs. It is worth emphasizing from the outset that the emerging processes and practices of automation and algorithmic processing have their own material and environmental presences (see for example Crawford, Citation2021). However they might be envisioned, algorithmic systems are, of course, embodied and instantiated (see Hayles, Citation1999, Citation2005). Data and algorithms have a very material presence, whether that be in the ‘dashboarding’ that gives them a visual form (Tkacz, Citation2022) or the corporeal interfaces that shape access and experience (Galloway, Citation2012) through to the cables, servers, and data centres that are involved in storage and retrieval (see for example Brevini, Citation2020; Hogan, Citation2015; Van Es et al., Citation2023). They are also embedded into the routines of everyday practices (Willson, Citation2017), including labour and organizational practices. The housing sector is in a time of transition, with developments in data analysis and automated decision making being integrated across the sector (see for instance Ferreri & Sanyal, Citation2022; Fields, Citation2022; Wainwright, Citation2023). This is a part of how, as Rose (Citation2017, p. 779) has observed, ‘cities across the world are being reconfigured by the deployment of digital technologies’. It is important to understand the materiality of these seemingly immaterial algorithmic processes as they shape decision-making. This is also important because the way these decisions are automated will in turn then shape decisions about housing allocations, that then impact on the distribution of populations and the ‘digital geographies’ of how spaces are inhabited (Bennett et al., Citation2023), which rental properties are available to people to rent and occupy (Maalsen, Citation2020; Parkinson et al., Citation2021), and even through to the structured dynamics of ‘invisible cities’ (Wyly & Holloway, Citation2002), data discrimination and homophily (Chun, Citation2021), or who is ‘on the edge’ (Colebrooke et al., Citation2023).

It is ‘amid the cacophony of noise around the big in big data’ that Amoore and Piotukh (Citation2015b, p. 360) ‘urge careful attentiveness to the work of the little analytics’. Moving beyond wider and more general ideas about big data requires attention to the close-up actualities. By focusing on decisions within the housing sector – the powerful yet little analytics of housing – we look more specifically at the materiality of algorithmic systems in relation to embodied human practices. We interpret this phrase to mean the smaller scale but direct and impactful ways in which data analytics are realized within specific instances. Indeed, our project has found that the housing sector has a variegated algorithmic landscape, with varying levels of automation and, crucially, different speeds with which organizations and parts of the sector are adopting algorithmic processes (this is something we are continuing to explore on the project, and have begun to outline the form and reasons for this variegation in Beer et al., Citation2023). Even with algorithmic and machine-based judgement the specifics of context are important for understanding the interventions made (see Roberge & Castelle, Citation2021, p. 18). Our focus is not just on the code and automated products, but on how people come to understand and engage with these systems. Here we concentrate on how participants themselves define or label aspects of their work as being manual. We focus on how practices can be categorized or labelled as manual in the context of algorithmic interventions. Here the distinction being made by participants is not between manual and non-manual (see for example Neal & Rettig, Citation1963; Smith et al., Citation1991), but between manual and automated or between manual and algorithmic. As such, this article identifies how the notion of the manual is deeply embedded within seemingly automated algorithmic decision-making processes. It is part of how actors make sense of these systems, or what Ruckenstein (Citation2023) has called ‘the feel of algorithms’. The article explores how the notion of the manual endures as a means for articulating the presence of the human actor, the limits of automation and the embodied practices of decision-making.

The broader project of which this article is a part focuses upon the integration of algorithmic systems within decision-making in the housing sector, with a specific focus on the societal impact of risk-profiling tools and other technologies deployed by lenders, landlords and others involved when people seek rental housing or mortgages. With ethical clearance granted by the host university, it incorporates 121 interviews with those working across different parts of the housing sector, especially with regard to granting private or social tenancies and mortgage lending – including private rental landlords (coded as PRS Landlord), social housing (SH), tenant referencing firms (TR), letting agents (agent), insurers (insurer), mortgage brokers (MB), lenders (Lender), credit assessment and software firms (Credit Firm), tenants (tenant) and professional membership or trade bodies (stakeholder). Initial explorations of the interview data, from both the raw transcripts and through NVivo coding, revealed recourse to the term manual – as such, this article is an analysis of the way that the term is used to describe practices, it is about how those acts are constructed in these accounts and through this term. From this observation, further direct searches were performed to gather together all uses of the word ‘manual’ within participants’ accounts of their systems, processes and practices. Due to the role being performed by the specific term ‘manual’, we focus only on that exact term in this article in order to bring out these specifics and to examine its particular role in the discourse and ordering of these practices. What we found was that the concept now plays an active and defining role in the demarcation and evaluation of human and embodied practices. Taking this argument further, in this article we identify ongoing attempts at valuing the manual - by which we mean the way that the manual is valued whilst also referring to the active valuing and evaluating of the manual that occurs as algorithms take on a greater role within practices and decisions.

The article begins by looking briefly at how the notion of the manual is used to demarcate or to mark out spaces and practices that are either not automated or not yet fully automated. We use this opening section to show how this notion is used to identify the degree to which a system is algorithmic. Following this broader application of the term, we then look at three specific ways in which the notion of the manual is operationalized within understandings of housing decisions. First, we focus on the manual as a form in need of input or completion. Second, we look at the manual as the checking of information or outcomes. Finally, we look at how the notion of the manual becomes a validating presence in the referral and approval of decisions. The article argues that the notion of the manual provides a means for demarcating human interventions within housing decision systems and that this presence is used to structure and validate the decisions as they are taken.

The manual as demarcation and the degrees of automation

We have indicated that the use of the term manual is applied in ways that enable demarcation between the human and the machine in general but also, more importantly, and more specifically, between an active decision and an automatic or algorithmic one. The manual is not just about physical labour and nor is it a way of simply separating out decisions into human or algorithmic, it has also now adapted to involve demarcating agency and cognition within the systems through which decisions are made. Indeed, this article indicates how the relations between types of agency are crucial in understanding the integration of algorithms in the sector – we identify this here as a key question to be addressed. In the case of this particular article, we have found that there exists a mixture of automation and human agency within decision-making systems, forming different combinations within different parts of the sector. The focus on the notion of the manual gives glimpses as to how these combinations are understood and handled, whilst leaving open the possibility of exploring in further detail this particular, to use Katherine Hayles (Citation2017) term, ‘cognitive assemblage’. In the housing sector, there is a highly variegated landscape of automation. What the notion of the manual does is to break this down by providing a way to define spaces, roles and involvements within these decision-making systems. The manual implies a materiality and an active presence that avoids the dissolution of such distinctions and instead marks out active human components or interventions within algorithmic systems. These sometimes also indicate future demarcations and areas in which automation is expected to be slower or where it is expected not to be applied at all.

This demarcation is also not fixed within these systems but changes over time. The valuing of the manual is not simply about demarcating spaces for protecting human involvement or intervention, it is also about the way that evaluation occurs and alters. This is, as Muniesa (Citation2012, p. 32) has suggested, to take a ‘viewpoint [that] shifts attention to valuation as an action’. We are approaching valuation as such an action, rather than as a fixed concept. Indeed, we cannot fully elaborate all possibilities of the concept of ‘value’ here, which we note is an area for further investigation and a complex topic in its own right, especially in relation to automation, instead we draw upon the distinction between value and values made by Skeggs (Citation2014) as a reference point and use this argument concerning the ‘co-constitution’ of the two to think about how, in the case of the application of algorithms within this particular sector, values will, as Skeggs put it, ‘haunt’ value. There are also questions created by automation about how valuation operates, especially in relation to work. Richardson (Citation2024) has explored how the automation of office infrastructure, including things like productivity tools, has implications for the valuation of work. Richardson (Citation2024, p. 8) develops the term ‘regimes of valuation’ to think about how ‘plural’ valuation approaches abound with automated infrastructures. Richardson (Citation2024, p. 2) argues that ‘it is this shift from automation targeting what work is done to how that work is done that raises the question of work valuation’. It is this question of valuation that we are also considering. Richardson looks at how that valuation of work itself is occurring through automation, whereas we are looking at how automation is itself subject to valuation as it becomes part of how work is conducted within the housing sector. Muniesa (Citation2012, p. 33) notes that ‘periods of unrest in valuation often open interesting opportunities for the questioning of available theories of value’. The integration of algorithmic systems may be creating a period of unrest, which alters value and may even create opportunities for thinking again about theories of value themselves.

The active valuing and evaluation of the manual that occurs around these algorithmic processes provides a means of explanation for the changing embodiments of algorithmic systems over time. The notion of the manual is used to stake out areas in which automation might advance, indicated by statements such as one participant in tenant referencing commenting that ‘that’s an area where you can take it as us slashing the manual labour that’s applied by utilizing technology’ (TR2). Even where there is a distancing occurring, in which the manual is presented as something to move on from, the notion is still facilitating demarcation. These are found in worries about errors, such as the observation that ‘you can store the wrong things in the wrong place when you do it manually, we’ve all made clerical errors with misfiled documents’ (Tenant 18). It is also found in concerns that such work is time-consuming and monotonous where ‘they have to input into the system, and then it’s a much more manual, laborious process for the agent’ (Stakeholder 3). Here the manual may not be valued as such, but it is still being used to demarcate the edges of automation and to evaluate human input.

The desire and extent of algorithmic input may vary, yet the term ‘manual’ still functions to demarcate on various scales. For instance, one participant in tenant referencing told us that for them:

it’s a very, very manual process at the moment. What we’ve actually just been doing is pulling together all of this manual process into a huge flow diagram. It was quite an undertaking because there’s so much stuff that we, on the process side, and systems side of things, didn’t have any visibility of. (TR8)

Here the flow diagram provides a visualization of the location and marked-out manual aspects of the tenant referencing process (for more on the development of automation in rental and tenant referencing in particular see Wainwright, Citation2023). The manual serves a labelling function within the process being accounted for within the diagram. The referencing system described here is also based on a scale of how manual a process is, in this case with it being described as ‘very, very’. A similar phrasing of ‘a very manual process’ (Stakeholder 11) was used elsewhere to emphasize the extent of automation.

This approach to viewing the manual on a spectrum or scale is something that came up elsewhere, indicating that it isn’t a simple demarcation process but that there is also a sense of degree of automation within these evaluations. The term is not just used as an either/or, but to think about the extent of automation within a system and to imagine a continuum of automation (and so providing a particular application of the wider continuum of algorithms described by Matzner, Citation2019, p. 129). For instance, another research participant working in tenant referencing indicated that:

There are different degrees of this when you really break it down and review it, so it’s certainly not going to be one day everything is manual, and this day a bunch automated. It’s going to be very slow. It’s going to be about largely, if we’ve checked people before or employers before, so that kind of thing. Having confidence in the employer is one area. (TR1)

It is not just which bits of the system remain manual, there is also a notion of the extent to which a system and its components are manual. This is understood as being about degrees of automation.

The degrees of automation are sometimes also approached in terms of the balancing between demarcated manual and automated processing. The blend of manual and algorithmic feeds into perceptions about the degree of automation. In this sector and in these systems, it was suggested that:

It’s always a balance to be sought. Some of the information you get from the automated processes is delivered so much more efficiently and so much quicker than you would doing it manually, and it’s of the same quality and that’s great. I just think in a lot of use cases it needs to be allied with a bit of judgement. (Stakeholder 1)

Parts of the process are demarcated as manual because of the properties that this is seen to retain or introduce. If judgement is required we see here how the manual is regarded as the means to achieve that – this is something we will return to when discussing both the manual checking and approval of processes. This input is retained even though, as it was put by another respondent in social housing, ‘it’s the manual bits that take the time to confirm the reference’ (SH1).

The manual can be demarcated within stages of such processes, with certain allocated stages being automated. In some instances, this can mean an action simply being removed from the category of the manual. It is the way the manual is used to outline the changing stages of the process that is crucial here. As is revealed in this account from an agent:

It removes all the stages of referencing that we used to do manually, so they do employment references, existing landlord references, which is basically what we used to do. Of course now they do credit checks as well, so it’s possible for agents to do those manually, or they need to get the tenants to do them for themselves. They do that, and then produce a report that looks nice and it gives us the ability to send that out to landlords, and it looks professional, and it looks independent as well, which we always think is quite important as well. (Agent 1)

Taking certain stages out of the category of the manual and placing them into the category of automation is illustrative of how the term facilitates notions of stages within a process and also how those stages can change. The manual doesn’t just demarcate within a system, it is also used to explain change in that system. The manual completion of forms was one important aspect of the changing forms of automation (as we will discuss in more detail in a moment). A participant involved in social housing observed how the automation is being applied:

means that the lettings officers don’t need to manually fill out the forms, so they will manually input the information about the applicant, the tenants. Then when we offer them the property, the offer letter is all automated, the tenancy agreement is all automated. So, that saves them time. (SH1)

Again here the stages of the processes are demarcated as manual or automated in their form. It is the manual filing of the form that we will explore further.

The results of these demarcations of the manual from the automatic is a sense of the mixed or hybrid nature of the systems this produces. This mix is then highly varied across the sector. As one respondent explained when asked about automated products, they replied that ‘there are still more traditional products out there … the one that we currently use is a bit of a hybrid, so it has more of a manual element in there, but then you’ve got a tiered basis’ (Stakeholder 1). This hybridity can shift with different balances or ‘tiers’ of automation or manual processing. The account is of shifting reliance on the manual.

In some cases the demarcation around the manual concerns certain types of tasks. As well as inputting, another example is that of adding narrative within the reporting stages. For instance, it was put to us that:

The more comprehensive report is more of a narrative, and I think they’re the ones that landlords traditionally are more comfortable with. They have the manual elements that directly get at an employment reference, a previous landlord reference, they evaluate those and they provide an evaluation, essentially, to a landlord and allow them to make their own decision. (Stakeholder 1)

The addition of narrative is seen as manual within these different degrees of automation here too. The requirement to be comprehensive and for something that fits with the conventions of processing expected by landlords means that the demarcated manual components are highlighted. The manual is something that is associated with particular value here – especially in facilitating the narrating of data. The manual is a concept that is deployed to build trust – with evocations of the virtues and type of trust placed in craft and in the handmade (Sennett, Citation2008, p. 104 & 117). As one insurer explained:

So we wouldn’t ever want to switch off that manual, human intervention because you’ll just lose a load of business. It might become efficient, but it’s not what we’re about. [name removed] the guy that runs the business, is always about being able to deal with everyone and make sure you can find a solution for everyone (Insurer 1)

The notion of the manual and the demarcation it affords generates perceived trust in the system (as well as perceived flexibility too). The manual has facilitated the automatic, by underpinning its use and qualifying it as appropriate and trustworthy. There are expectations concerning what is automated, with algorithmic processing being more efficient whilst also being perceived to create possible problems around trust.

The values associated with the notion of the manual and the role that this notion performs in demarcating human and algorithmic roles or aspects within housing sector decisions need to be unpicked carefully. To develop these points further and to think in more direct terms about the valuing of the manual, we take three particular types of specific demarcation that are associated with the manual. These allow the specific and detailed aspects of the demarcation of embodied practices within algorithmic decision-making processes to be developed further. In his genealogy of data systems, Koopman (Citation2019, p. 12) identifies ‘three kinds of fastening enacted by informational formats’. As he elaborates, there is:

The standard taken from a common technical definition used in information systems analysis according to which information handling can be parsed into three phases: data collection and storage (the input phase), data analysis and augmentation (the processing phase), and data dissemination and reproduction. (the output phase)

Koopman draws upon this standard idea of information systems to frame his exploration of data ‘fastening’, which refers to how people are situated and fastened within data formats. These three phases may seem linear from input to output, ‘and yet also form a loop or a circuit that sets enabling conditions for information to amplify itself’ (Koopman, Citation2019, p. 12). We note how the manual can be integrated, into different configurations, during these three phases. The form is at the input stage, the check is at the processing stage and the approval is at the output phase. The three demarcations performed by the manual map onto these three phases, revealing how the notion of the manual can be incorporated at each phase. We turn first to the manual as form.

The manual as form: at the input phase

The first application of the notion of the manual comes with the form, and often with the manual completion of some type of paperwork. Of course, these often don’t take the form of actual paper, yet the notion of manually completing or inputting information into the form remains. Here the actor is adding information into the system, adapting to categories, narrating or answering. The activity of inputting often falls within the boundaries of what is understood to be manual. The manual inputting of information into forms thus represents a limit of algorithmic processing. The form represents an object upon which manual effort is exercised or applied. Simple inputting can appear to be of minimal importance, yet Koopman (Citation2019, p. 12) argues, it is a central part of how ‘Infopower is exercised through the quotidian work of formatting’.

In the case of tenant referencing, for instance, references themselves become the form that requires information:

I just need to manually progress the referencing. I’ve manually updated the reference, but it might take again a few minutes until it updates here. While that’s happening, I will just go through other sections. We have another section which is the document section, and this is where the agent can find the documents that have been signed so far, but also upload copies of the documents which then need to be served to the tenant before they can move in. (TR6)

The image here is of the manually updated reference and the checking of other documents (we will return to the process of checking in the following section) – even though the system checked or updated is itself digital. The updating and progressing of the reference are demarcated as manual actions. Similarly, another participant working in social housing was asked about automation and responded:

No, no, it’s all done manually. So they complete the form. As I said, if there are high risk answers for more than two elements, then it comes to [named member of organisation], myself, to assess, and we will make the decision as to whether or not that the property can be offered or not. (SH1)

Here the form also offered one means for articulating the manual as practice. The form represents the activity for which manual effort is required. This manual labour is on the part of the applicant, in terms of completing the form, but then this is also assessed manually for the decision to be made.

In general terms, the balance between manual and automatic is something that is actively being sought and explicitly managed, especially with changes being introduced by new products and systems. This can also bring out concepts of old and new forms to be completed. One participant explained this within an account of the move away from paper-based approaches, detailing how:

there is a lot of activity going on in terms of digitisation, but I think it’s a really, at the moment anyway, seems to be more around the processes because, internally, processes are so manual, and paper-based. We’ve had lots of presentations from, whether it’s digitising that process, digitising even the connection between their internal systems is still difficult because, well, they’re on old systems and they won’t have that connectivity between maybe the system that they engage with customers or mortgage intermediaries. They’re really trying to overcome those technological barriers as well, which is starting to happen. I don’t know if that, as well, once they start to really make efficiencies there, whether you’ll start to see people more turn their attention to pulling other things, but my impression is in terms of … processing, it’s been relatively static in the mortgage market. (Stakeholder 2)

The processing of forms is one way in which the manual is articulated in practice. The processes, as this respondent put it, are both ‘manual’ and ‘paper-based’. This respondent gives a sense of change, with the barriers to the changes in these systems being identified as associated with the format of the forms. Within this, the notion of change is attached to the manual and the extent of manual input within the processing of decisions. Automation equates to the reduction of manual input. Partly this is because, they explain, of the tangible difficulties of creating fully digitized records, never mind the problems of fully automated systems in practice. The scale of all those forms to complete is one image that is projected. Manually inputting or managing at that scale was identified, through the concept of the manual, as being an issue in some instances – the algorithm was understood to be able to handle such scale.

These types of barriers, of course, are expressed in terms of cost with one suggestion being that ‘the kind of thing you can access very cheaply … It’s likely to be more manual’. (Stakeholder 1). Aspects of the system are more likely to be embodied and manual because of cost in this case. Here the manual becomes a means by which variations in systems can be understood and explained. It also becomes the site of the limits of algorithmic developments and change. These types of variations can occur within systems as well as across them. The manual provides a way of articulating the adaptation of the form to the situation. One participant working in social housing spoke of the flexibility of the manual by explaining that:

If somebody didn’t want to do open banking, we would still need them to provide two months’ bank statements. So, they could either send them in, and we’d do it a manual way … We try and take a flexible approach, and try and do a manual income and expenditure. (SH7)

Referring to Open Banking, which is a means of affording digital or automated access to an individual’s bank data for assessment purposes (for discussions of Open Banking see Ciocǎnel at al., 2024; He et al., Citation2023), the manual is demarcated in terms of the completion of processing using means that bypass the restrictions of more automated systems. Alongside the form, it is the way that information, references and decisions are then manually checked that is important in these notions of embodiment.

The manual as check: at the processing phase

Inputting into forms is one side of this process, the input stage. We also find the manual at the processing stage. We have seen hints already that information processing and decision-making often require some form of manual checking. The above discussions of the use of forms indicate how the processed form can require checking once completed. In this case, the notion of the manual is used to integrate or maintain human oversight into decision-making. The manual here is about checking information, accuracy, plausibility or any other feature of the basis for decisions. To return to a point with which we opened this article, according to Amoore and Piotukh (Citation2015b, p. 346), ‘it is the gathering and reading that form part of the work of the little analytics’. Here, in the little analytics of housing, we see exactly this. When looking close-up at the specifics of practice we see gathering and reading as key aspects of how these systems are organized around algorithmic intervention.

To explore this type of manual checking, let us take one example from an account provided by a participant operating within tenant referencing. In terms of the tenant referencing process, the checking part of the process, this respondent described, ‘at the moment it’s all manual, and very much up to the referencers’ due diligence to spot things’ (TR8). The ability to manually check and to spot is important here. The referencer is engaged in a manual process in which they are checking to, as it was described, ‘spot things’. The manual here refers to this checking of the detail to spot anything important that might otherwise have been missed by the algorithm. The vision here is of the manual as an important presence for checking details – making it also obdurate to the expansion of the algorithm. In conclusion, this participant added:

I think it’s always going to be the way in which you’re gathering that information which is where you’re looking to decrease time to get that information, or accuracy of that information. I think because that works so well, and if you’ve manually got someone picking up the phone and speaking to these individuals, there’s not much better that you can do. (TR8)

Over time inputting may become automated, but the checking and verifying is understood to be more enduring and less likely to move away entirely from the manual. Because of the need for assuredness in the information, the manual is seen as an ongoing necessity. The manual is required to check and ensure accuracy. The manual is seen here as more reliable – someone ‘manually … picking up the phone’ is considered, for instance, to be a trusted means of verification. This participant involved in tenant referencing went on to explain how, once the forms are complete:

All of this information at the moment goes to a referencer in our [named city] office which is a very manual process. They pick all this information up, verify it with lots of phone calls and emails, and something that we’re currently exploring at the moment is, how can we better verify a tenant’s affordability, mainly. (TR8)

The manual check involves the individual referencers verifying information, sometimes on the phone. The phone call is an example of one key type of manual intervention that we aim to explore further in future work.

Above the manual verification is described as central to ensuring the decision. The manual demarcates validation exercises through material and embodied practices. Adding to this, this same participant speculated that in tenant referencing in the future:

We’ll still have a referencer, so a manual person looking over that data and saying yes, we definitely feel confident in that. So there’s still always going to be some sort of manual intervention to make sure we’re interpreting that data correctly, but actually getting that validation, and that back and forth, isn’t going to take us long. (TR8)

We’ll still have a referencer, so a manual person looking over that data and saying yes, we definitely feel confident in that. So there’s still always going to be some sort of manual intervention to make sure we’re interpreting that data correctly, but actually getting that validation, and that back and forth, isn’t going to take us long

The manual, it would seem, is a notion that is likely to endure. The weight attached to its importance within these processes gives it obduracy. It is the confidence that the notion of a manual check brings that is envisioned as likely to be sustained. We’ve looked at one participant in detail in this section, but these themes and this interest in the manual check are of wider concern and import.

This type of focus on the retention of the manual for its value in verification, in particular, is found elsewhere and is articulated in similar terms. Another participant working in tenant referencing discussing the question of automation also concluded:

But you still need … the human to have a look at things like documents, potentially, in terms of their contract of employment, or you need to ring the employer and have a chat with them, too. That’s probably the best example. Now, in the past, before open banking technology was around, you would have to do all of that manually, so potentially, as a worst-case scenario, you might be looking through 12 months of bank statements (TR4)

The manual as form, as discussed in the previous section, transfers at the processing stage into manual checking. There are changes in the amount of information being checked, which hints at some of the changes brought about by the ‘new data relations’ (Kennedy, Citation2016) within such organizations, but with reduced manual intervention. The key point here is the emphasis on still requiring human input and still needing some manual checking, if not on the same scale as in the past.

The manual check occurs in other places too, sometimes explicitly and sometimes more implicitly in accounts of the processes undertaken – this reflects the housing sector’s variegated algorithmic landscape that we have mentioned at the outset. When reflecting on how things might be done elsewhere another participant working in tenant referencing turns to the notion of the manual:

I’m not sure exactly how they do it. I believe it’s a manual process. It’s automated but yes, effectively it’s manual. So employers are contacted every time to just establish, yes, this person is employed, yes, this person is self-employed, they’re a contractor, or they’re on a short-term time-limited contract. (TR7)

There is a little uncertainty, not about the process perhaps but about how to separate the manual from the automated. We shouldn’t read too much into a momentary hesitation, yet this may indicate a blurring of the boundaries in practice and the capturing of change. Yet the manual is still directly attached here with the checking process of making contact to verify details. When another respondent working in social housing was asked about automation (for an account of social housing and electronic delivery see Pleace, Citation2005), they were clear that it was ‘Nothing as fancy as that. No. It’s all very, our right to rent checks are quite manual, really, aren’t they? The customer updates it, and then uploads it to the website, and then we check it for them’ (SH7). As the participant corrects the visions of automation it is made clear that checking remains as a manual process. The completed digital form is checked manually.

In other instances where automation is more advanced, in this case in mortgage advice, this manual checking is still retained. As it was explained to us:

When they’re doing the mortgage advice process, so the AI tool will be helping them as the customers answers each question it will be computing to the next question until it’s come to an idea about that customer, but then our members don’t want it to go any further. They’ll still want to then manually take the advice process. Take their bank details, and use their own risk scoring internally. (Stakeholder 2)

There are expectations, this suggests, about the inclusion of manual checks. The demarcation of the manual is important in the sense of facilitating trust in the system. Thus the term manual serves to indicate what is retained and to serve expectations. Such perspectives are illustrative of the valuing of the manual that is occurring. There is an evaluation of the importance of the integration of manual checks in this case.

This importance of the manual in facilitating trust takes other forms elsewhere. The manual can literally generate value in these cases too, with the retention of manual checking being regarded as a more premium service. On this point one insurer offered the following comparison:

They’ve asked for an employer reference, and then there was no checks aroundit, whereas one of the really good companies we work with will have a flag to go, this is like a Yahoo account or a Gmail account, and they will then put a manual process in to check that company does exist, and phone them up, check Companies House, etc. So that’s where you get gain in the system. (Insurer 1)

It is not just the presence of the manual generating value here, instead, this takes us back to the idea of demarcating to manage the balance between the manual and the algorithmic. The evaluation of the manual here is captured in the phrase ‘you get gain’. This returns us to understandings around the degree of automation. It is the blending of these two, with the automated flagging and then human manual checking, that is presented as the ideal. It is the notion of the manual that enables these types of articulations of the blending of human and machine and also that enables the value and valuing process to then occur from that set of demarcations.

The way checking works in practice, as well as what is meant by manual, is inevitably more complex when specific practices are examined even more closely. This additional depth is revealed by an account from a research participant working in social housing, who pointed out that:

We’ve had it in place for a number of years, so even before we got this automated system. Let me share with you the story first of all. What we used to do is just a manual check. When I say a manual check, we used to fire it off into the [named credit referencing agency] system and just type in a name and an address, and then we’d get a credit report back. We would manually look at that to decide whether or not that is somebody that needs to be going through the threshold. When it comes to social housing, it’s a different kettle of fish when we go on to the private market rent in a second, but what we’ve found as a social landlord is that it’s a good automated system and we can key our lettings policy into, when I get the name for it, whatever it’s called…. That would give us an accept, refuse, or defer as an outcome once we’ve typed that in. We’ll look at it against our parameters, and the parameters, one of the key ones at the moment, is a £5 disposable income, but you can set that wherever. During the cost-of-living crisis, we’ve had to keep bringing that down because people have less of a disposable income and it’s just flagging more to fail. (SH 2)

Through the discussion of ‘thresholds’ and ‘parameters’ we can see how the checking operates against a frame of reference and notional limits. These parameters alter in terms of where they are set. The check remains in some sense manual even if the automated system is incorporated into the process – this is required for trust to be placed in the system. Here, in the narrative of the above account, we see first how the systems are subject to change and also how different types of and levels of automation shift over time. The manual is not fixed. Nor is it a fixed concept. What is meant by the manual might vary. It is actively being reworked within these accounts. In this case, the respondent is working out within the passage what the extent of automation is and what they specifically mean by ‘manual check’. Third, this account concludes with an insight into how manual interventions are needed, following manual checks, to alter parameters to produce certain outcomes. This is a manual check over the functioning and output of the automated system, with the automated selections then being changed in light of those checks. Manual checking of different types, along with the notion of manual checking, is subject to change and to ongoing valuing processes.

The manual as approval: at the output phase

Finally, the notion of the manual is used to demarcate a final or closing stage within the decision-making process in which the human intervention comes in the form of approval of an outcome. The manual is often incorporated at the output stage where the final decision is made. With earlier stages being partly handed to the algorithm and with inputting being done through combinations of manual and automated processing, the final stage of the decision, the final approval, appears to often be made through a manual intervention. This extends the checking aspect, taking it to its final output point in which the decision or outcome is finalized. The important feature of this concerns timing, in which the decision is referred to a human and to the practice demarcated as the decision. Such referrals for manual decision and approval can represent most of the manual labour. For example, one insurer concluded that: ‘Yes, absolutely. That’s the bulk of our workload, is the manual referrals’ (Insurer 1). That same participant explained how ‘we default back to our standard terms and conditions that they need to meet. So basically, going back to there must be the ID, there must be the affordability, there must be the clean credit check. If they don’t, then it’s a manual referral to us and we’ll do it’ (Insurer 1). The manual is both check and then referral for approval. The output comes with a closing manual intervention. It is possible that it is the oversight provided by retaining the human at the final stage that enables the integration of algorithms at the earlier stages of inputting and processing.

These referrals mean that the manual action is embodied in the act of the approval. The approval or final decision is manual and is therefore marked out as human rather than algorithmic. The final output is a product of the human actor overseeing the various blends of algorithmic input. Whereas in the previous section it was the eye for accuracy that was performed manually in the act of checking, here it is the flexibility and oversight that is offered by a final human decision that is manual. In one case, for instance, it was explained how ‘if the referencing algorithm doesn’t fit the tenant’s situation, then your letting agent has to phone them up and ask them to manually approve it’ (Stakeholder 6). The manual is a human approval process here. The final decision is demarcated as manual. In such cases, the outcome is a type of ‘manual intervention’ (SH2). A manual intervention is made in this case to ensure the particular circumstances and contexts of individual cases are considered fully and adapted to – this is in contrast to the seeming inflexibility of the purely algorithmic decision – indeed, some there were instances where smaller lenders explained how the manual decision was a key part of how they understand their own value and present that value. This emphasis is captured in phrases like: ‘making a manual decision about whether or not we do that’ (SH2). Illustrating the focus on the flexibility of manual interventions at the approval stage, it was suggested that the existing track record impacts ‘how many manual interventions that a customer gets’ (SH2).

That flexibility for human intervention or decision-making was emphasized elsewhere. One stakeholder gave two examples to illustrate why a manual intervention might be needed to bypass the algorithmic decision:

‘Then you might have other assets. You might have a house, so in my… My parents-in-law have two houses. Although they rent themselves, they have two houses that they rent out. You then have a form that doesn’t really allow you to declare that properly. Effectively, at the end of the day… Oh yes, this is the other one. If you can’t prove one bit of your income, you fail the referencing, even though what you’ve already proved gets you well over the bar. What happens is, the letting agent phones up the referencing company and says, “Can you do a manual approval?” and they manually pass it. It circumvents the algorithm because the algorithm doesn’t fit that particular situation’. (Stakeholder 6)

The manual is used here to outline how the algorithm might be circumvented in the decision-making process. The manual is equated with flexibility and an ability to understand personal narratives – which is placed in contrast to the algorithm’s rigidity. There is an implicit defence of the human and the presence of a type of ‘culture of maintenance’ (Pasquale, Citation2020, p. 32) in which the algorithm complements rather than replaces the human within decision-making. The manual approval is a means by which the algorithm is overridden. The manual is valued here in terms of flexibility and also in terms of the possibility of approval or making a decision that is not seen to be dictated by algorithmic selection or by the limits of certain categorical restrictions or missing data.

As this would suggest, the notion of the manual is incorporated into accounts of the approval of decisions as a means of control over the factors involved. This was described in terms of the ‘decision bit’:

‘Yes, well, I think that would be saying for our members, even where they’re doing their automating elements, by which I probably mean just you won’t have to manually input it in. It will flow more seamlessly, but then actually that’s the admin bit. In terms of the decision bit, I think there’s still people want to be in control of that. That makes it almost sound like that will remove a hell of a lot of time by removing the admin bit. That person is maybe making a choice, you can have all that information that may have taken days, if not weeks and months to collate otherwise.‘ (Stakeholder 2)

The decision is seen as something to be kept manual. The manual intervention equates with the retention of control over a decision. The ‘elements’ that are automated, this suggests, are the inputting elements. Returning to the degree of automation outlined earlier, we can see here how the manual is used to separate the human from the automated within the decision-making, and also, alongside this, how the manual is valued in terms of facilitating a control over decisions. This oversight then, of course, has implications for housing outcomes – in some instances the manual approval was seen as a way to check that algorithmic profiling is fair.

Conclusion

‘Information’, as Katherine Hayles (Citation1999, p. 49) has observed, ‘cannot exist apart from the embodiment it brings into being as a material entity in the world; and embodiment is always instantiated, local and specific’. We have looked at this embodiment of information as local and specific to the housing sector. It is when algorithms and automation are already ordinary parts of social life – with an established ‘algorithmic life’ (Amoore & Piotukh, Citation2015a) – that notions like the manual take on new roles and have a new significance. We have shown here how the notion of the manual is a part of the ‘fastening’ (Koopman, Citation2019, p. 12) process of data systems. Phrases like ‘very manual process’ and even ‘very, very manual process’ are indicative of the evaluation of the manual in practice. They indicate the extent of the degrees of automation as defined by those working with these systems – which provides insights into the ‘intersections of … labour with with the rapidly changing socio-spatial relations brought about by new forms of … automated life’ (Bissell & Del Casino, Citation2017, p. 440). Andrejevic (Citation2020, p. 18) has argued that with the ‘offloading’ of human decision-making processes onto automated media systems, it will be important to identify the tendencies of these systems’. Looking at these types of tenancy and lending processes, the notion of the manual reveals something of the tendencies inherent in these emerging decision-making systems within the housing sector, especially with regards to how embodiment is demarcated.

We have explored here how the manual is a part of how algorithmic systems and processes are understood and organized. The notion of the manual is active in demarcating aspects of the decision-making process in contradistinction to the automatic or the algorithmic. We have used the central focus on valuing the manual to enable us to think here both of how value is attached to the manual and also how the manual itself is evaluated and assessed as part of the changing forms of these processes. Focusing upon the notion of the manual enables an understanding of the demarcation and evaluation of embodiment within algorithmic decision-making processes. By this, we do not simply mean that individuals or organizations hold onto a sense of the manual because they value it, although we are suggesting that this is sometimes the case. Rather, what we are suggesting is that the valuing of the manual is an ongoing process of active demarcation, assessment and validity. The valuing of the manual as an active form of assessment and reassessment of its purpose and position within algorithmic developments.

Indeed, the manual takes on a stronger purpose when it is used to demarcate spaces and practices within advancing forms of algorithmic automation. The notion of the manual has more work to do where there is a clear alternative to it. We have noted here three ways in which the manual is situated within these systems and within these demarcations. The first was the manual as form, or the practice of completing or updating some sort of form to provide information or to allocate or categorize. The second was the manual as check. This required the processes of assessment and evaluation to be conducted manually and with direct or predominantly human intervention. The manual act is involved in checking a decision or checking the information upon which a decision is based. Third and finally, we looked at the manual as approval. Here the manual is attached to the validation, confirmation and legitimation of a decision. The manual here is an intermediary step or intervention through which approval is gained or recorded. These three closely related forms of the manual are often integrated with one another and are active within the demarcation of roles and divisions within the systems and processes through which housing decisions are made.

Acknowledgments

The CODE ENCOUNTERS project was funded by the Nuffield Foundation, but the views expressed are those of the authors and not necessarily the Foundation. We are thankful to the reviewers for their comments on an earlier version of this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the Nuffield Foundation [Code encounters].

References

  • Amoore, L., & Piotukh, V. (Eds.). (2015a). Algorithmic life: Calculative devices in the age of big data. Routledge.
  • Amoore, L., & Piotukh, V. (2015b). Life beyond big data: Governing with little analytics. Economy and Society, 44(3), 341–366. https://doi.org/10.1080/03085147.2015.1043793
  • Andrejevic, M. (2020). Automated media. Routledge.
  • Appadurai, A. (1996). Modernity at large. University of Minnesota Press.
  • Beer, D., Wallace, A., Ciocanel, A., Burrows, R., & Cussens, J. (2023). Automation hesitancy: Confidence deficits, established limits and notional horizons in the application of algorithms within the private rental sector in the UK. Information, Communication & Society, Advance online publication. 1–16. https://doi.org/10.1080/1369118X.2023.2264954
  • Bennett, K., Gardner, Z., & De Sabbata, S. (2023). Digital geographies of everyday multiculturalism. Social & Cultural Geography, 24(8), 1458–1477. https://doi.org/10.1080/14649365.2022.2065699
  • Bissell, D., & Del Casino, V. J. (2017). Whither labor geography and the rise of the robots? Social & Cultural Geography, 18(3), 435–442. https://doi.org/10.1080/14649365.2016.1273380
  • Brevini, B. (2020). Black boxes, not green: Mythologizing artificial intelligence and omitting the environment. Big Data & Society, 7(2), 1–5. https://doi.org/10.1177/2053951720935141
  • Bucher, T. (2017). The algorithmic imaginary: Exploring the ordinary affects of Facebook algorithms. Information, Communication & Society, 20(1), 30–44. https://doi.org/10.1080/1369118X.2016.1154086
  • Chun, W. H. K. (2021). Discriminating data: Correlation, neighbourhoods, and the new politics of recognition. MIT Press.
  • Ciocǎnel, A., Wallace, A., Beer, D., Cussens, R., & Burrows, J. (2024). Open banking and data reassurance: The case of tenant referencing in the UK. Information, Communication & Society, Advance online publication. 1–16. https://doi.org/10.1080/1369118X.2024.2310481
  • Colebrooke, L., Leyshon, C., Leyshon, M., & Walker, T. (2023). ‘We’re on the edge’: Cultures of care and universal credit. Social & Cultural Geography, 24(1), 86–103. https://doi.org/10.1080/14649365.2021.1921244
  • Crawford, K. (2021). Atlas of AI: Power, politics and the planetary costs of artificial intelligence. Yale University Press.
  • Ferreri, M., & Sanyal, R. (2022). Digital informalisation: Rental housing, platforms, and the management of risk. Housing Studies, 37(6), 1035–1053. https://doi.org/10.1080/02673037.2021.2009779
  • Fields, D. (2022). Automated landlord: Digital technologies and post-crisis financial accumulation. Environment and Planning A: Economy and Space, 54(1), 160–181. https://doi.org/10.1177/0308518X19846514
  • Galloway, A. R. (2012). The interface effect. Polity Press.
  • Haraway, D. (1991). Simians, cyborgs, and women. Free Association Books.
  • Hayles, N. K. (1999). How we became posthuman. University of Chicago Press.
  • Hayles, N. K. (2005). My mother was a computer. Chicago University Press.
  • Hayles, N. K. (2017). Unthought. University of Chicago Press.
  • He, Z., Huang, J., & Zhou, J. (2023). Open banking: Credit market competition when borrowers own the data. Journal of Financial Economics, 147(2), 449–474. https://doi.org/10.1016/j.jfineco.2022.12.003
  • Hogan, M. (2015). Data flows and water woes: The Utah data center. Big Data & Society, 2(2), 1–12. https://doi.org/10.1177/2053951715592429
  • Kennedy, H. (2016). Post, mine, repeat: Social media data mining becomes ordinary. Palgrave Macmillan.
  • Koopman, C. (2019). How we became our data: A genealogy of the informational person. The University of Chicago Press.
  • Ladewig, R., & Schmidgen, H. (2022). Symmetries of touch: Reconsidering tactility in the age of ubiquitous computing. Body & Society, 28(1–2), 3–23. https://doi.org/10.1177/1357034X221097068
  • Maalsen, S. (2020). ‘Generation share’: Digitalized geographies of shared housing. Social & Cultural Geography, 21(1), 105–113. https://doi.org/10.1080/14649365.2018.1466355
  • Matzner, T. (2019). The human is dead – long live the algorithm! Human algorithmic ensembles and liberal subjectivities. Theory, Culture & Society, 36(2), 123–144. https://doi.org/10.1177/0263276418818877
  • Muniesa, F. (2012). A flank movement in the understanding of valuation. The Sociological Review, 59(2 supplement), 24–38. https://doi.org/10.1111/j.1467-954X.2012.02056.x
  • Neal, A. G., & Rettig, S. (1963). Dimensions of alienation among manual and non-manual workers. American Sociological Review, 28(4), 599–608. https://doi.org/10.2307/2090075
  • Parkinson, S., James, A., & Liu, E. (2021). Luck and leaps of faith: How the digital informal economy transforms the geographies of shared renting in Australia. Social & Cultural Geography, 22(9), 1274–1290. https://doi.org/10.1080/14649365.2020.1723134
  • Pasquale, F. (2020). New laws of robotics: Defending human expertise in the age of AI. The Belknap Press.
  • Pasquinelli, M. (2023). The eye of the master: A social history of artificial intelligence. Verso.
  • Pleace, N. (2005). The shaping of electronic service delivery. Information, Communication & Society, 8(4), 524–541. https://doi.org/10.1080/13691180500259194
  • Richardson, L. (2024). Automated office infrastructures and the valuation of work. Environment and planning D: Society and space. Advance online publication. Environment and Planning D: Society and Space. https://doi.org/10.1177/02637758231218799
  • Roberge, J., & Castelle, M. (2021). Towards an end-to-end sociology of 21st-Century machine learning. In J. Roberge & M. Castelle (Eds.), The cultural life of machine learning (pp. 1–30). Palgrave Macmillan.
  • Rose, G. (2017). Posthuman agency in the digitally mediated city: Exteriorization, individuation, reinvention. Annals of the American Association of Geographers, 107(4), 779–793. https://doi.org/10.1080/24694452.2016.1270195
  • Ruckenstein, M. (2023). The feel of algorithms. University of California Press.
  • Sennett, R. (2008). The craftsman. Yale University Press.
  • Skeggs, B. (2014). Values beyond value? Is anything beyond the logic of capital? The British Journal of Sociology, 65(1), 1–20. https://doi.org/10.1111/1468-4446.12072
  • Smith, C., Knights, D., & Willmott, H. (Eds.). (1991). The non-manual labour process. Macmillan.
  • Taylor, C. (2004). Modern social imaginaries. Duke University Press.
  • Tkacz, N. (2022). Being with data: The dashboarding of everyday life. Polity Press.
  • Van Es, K., van der Weijden, D., & Bakker, J. (2023). The multifaceted and situated data center imaginary of Dutch twitter. Big Data & Society, 10(1), 1–16. https://doi.org/10.1177/20539517231155064
  • Wainwright, T. (2023). Rental proptech platforms: Changing landlord and tenant power relations in the UK private rental sector? Environment and Planning A: Economy and Space, 55(2), 339–358. https://doi.org/10.1177/0308518X221126522
  • Willson, M. (2017). Algorithms (and the) everyday. Information, Communication & Society, 20(1), 137–150. https://doi.org/10.1080/1369118X.2016.1200645
  • Wyly, E. K., & Holloway, S. R. (2002). Invisible cities: Geography and the disappearance of ‘race’ from mortgage-lending data in the USA. Social & Cultural Geography, 3(3), 247–282. https://doi.org/10.1080/1464936022000003523