Autopoiesis through agency in virtual reality nonfiction

ABSTRACT

Documentary filmmakers are gradually embracing immersive media to create novel Virtual Reality Nonfiction (VRNF) content. Over the past twenty years initial experimentation in this new medium has brought forward numerous linearly structured 360° documentaries that maintain a close link to traditional documentary modes. More recently, we have observed a shift from the relatively passive 360° cinema towards more open-world, non-linear, game-like interactive experiences that challenge traditional definitions of the documentary genre. Volumetric world-building techniques provide nonfiction creators with additional tools that afford ‘viewer-users’ spatial and interactive agency, leading to a heightened autopoietic realisation of the storyworld. VRNF creators have the potential to allow their viewer-users enhanced control over framing, temporal ordering of the plot and spatial unfolding of the diegetic world, thus inviting them to become actual co-creators of a deeply personal and personalized experience. This article addresses how VRNF may go beyond the mere ‘documentation’ of people, places or past events that existed in a pre-filmic reality and provide viewer-users through augmented agency a unique present-tense autopoietic experience that pushes the boundaries of traditional 2D documentary.

KEYWORDS:

• Virtual reality
• 360° cinema
• documentary
• cognition
• autopoiesis

Introduction

Early Virtual Reality Nonfiction (VRNF) productions have attempted, with limited success, to mould traditional linear narrative structures of documentary film on to Virtual Reality’s (VR) novel storytelling affordances, which are primarily characterized by spatiality, non-linearity, presence and agency. This is particularly evident in 360° VRNF, for example, Clouds over Sidra (Arora and Milk 2015) and The Fight for Falluja (Solomon 2016) make use of a linearly structured voice-over and chapters that lead their viewers through the unfolding narrative. Yet, VR’s novel affordances have the potential for unique autopoietic experiences, in which viewers, now more like users – or better ‘viewer-users’ – can actively influence and shape the simulated world and as such, create not only a personalized but also a deeply personal narrative. It is personalized, as the narration and the narrated are both shaped through the viewer-user’s agency, and deeply personal, as the heightened sense of presence inherent to immersive media may result in viewer-users’ more subjective and therefore intimate experience of the unfolding storyworld. We use the term viewer-user in the context of VRNF, as this represents a shift away from the passiveness of a mere ‘viewer’ but not yet allowing for the active and continuous input required by a fully empowered ‘user’.

While documentary makers’ creative routines, solidified in linear narrative structures and mediated storyworlds, may limit VRNF’s autopoietic potential, creators of VRNF experiences are beginning to embrace novel technological developments, and by that move away from the mere documentation of pre-filmic events towards eliciting truly active user-centric engagement. Traditionally, a ‘storyworld’ is the result of the cognitive action of the spectator who ‘builds up some version of the diegesis, or spatio-temporal world, and creates an ongoing story (fabula) occurring within it’ that is instigated by the narrative techniques utilized in an audio-visual work (Bordwell 1989, 8). In VR cinema, however, these narrative techniques shift away from being a highly mediated but, in their meaning-making affordances, rather constraining practice and place greater narrativizing potential on the viewer-user. In cognitive terms, the action of the viewers’ mental meaning-making is extended in VR cinema by the viewer-users’ concrete construction – personalized actualizations of the possibilities that a pre-determined but sandbox-like storyworld allows for, in which a co-creation option then results in a more personal experience and thus meaning-making. Indeed, in an attempt to create a first manifesto for VR cinema, Janet H. Murray (2016) states that 360° VR cinema should not be confused with a ‘film to be watched’, but rather a ‘virtual space to be visited and navigated through’. Furthermore, she adds that VR cinema should not be thought of as a ‘movie without a frame’ but rather as an embodied experience. Thus, when VRNF is centred around the viewer-user’s embodiment, i.e. a feeling of being physically present, they are afforded with agency. Through the levels of agency introduced by VR systems, viewer-users enter a complex interrelationship with the diegetic world and the underlying programming of the VR application. We consider this interrelationship an autopoietic system that is comprised of individual parts and is self-perpetuating, giving rise to an autopoietic mode of VRNF. This idea resonates with Aston and Gaudenzi’s (2012, 126–128) modes of interactive documentaries (i-docs): the conversational mode (the user explores 3D worlds as if ‘in conversation’ with the computer), the hypertext mode (the user is afforded the ability to ‘explore’ by selecting different options), the participative mode (the user participates in the creation of the artwork) and the experiential mode (that combines the virtual and the real in a hybrid space). Whereas these categorizations are useful for understanding i-docs, Virtual Reality operates on a distinct representational level due to a perceived dissolution of the screen-user boundary. Furthermore, agency (in VRNF), as opposed to interactivity (in i-docs), comprises the intentional actions that are required to make sense of the storyworld and potentially alter it in a meaningful way. Interactivity can be considered as more routine actions, such as point-and-click operations on a computer screen that do not necessarily incorporate the purposefulness of actions associated with agency (Murray 1997; Ryan 2015). Thus, we may argue that a potential autopoietic mode of VRNF is distinct, as it may combine and go beyond elements of i-doc modes to create a truly autopoietic system. Another useful starting point for understanding autopoietic VRNF is Gaudenzi’s concept of ‘Living Documentary’, which refers to ‘an assemblage composed by heterogeneous elements that are linked through modalities of interaction. It can have different levels of autopoiesis and can be more or less open to transformation.’ (Gaudenzi 2013, 26). Yet, Adjani Guerrero Arumpac (2020, 121) sees the use of the term ‘Living Documentary’ as finite due to the technological restrictions of interactive documentary, providing limited agency and possessing a curated structure. She prefers the term ‘regenerative documentary’ as ‘the cognition of an autopoietic living system as it interacts and transforms with its environment’ (113). Regenerative documentary interacts with distinct ‘nodes’, such as the involved collectives, the audience and the technology in an ever-evolving feedback loop (Arumpac 2020, 129). Here, however, we delineate an autopoietic VRNF system as comprising of the continuous exchange between the viewer-user, the technology and, in possible future multi-user experiences, the protagonists.

In this article we will attempt to adapt the term autopoiesis¹ to VRNF, as the ability of viewer-users to actively contribute, via different levels of agency, to the creation (self-perpetuation) of a storyworld (diegesis) through introducing a degree of randomness within a closed system (the pre-defined limits of the experience) that is ephemeral (non-repeatable) and personal (it cannot be shared).

The emergence of agency in VRNF

The idea of being able to capture the world around us in 360° is not new. Already in the 1970s, Margaret Mead (1975, 9) envisioned the 360° camera as an ideal tool for preserving visual material of disappearing cultures. She suggested then that, in the future, a camera equipped with a 360° lens could be set up in a village square and record footage without affecting people’s behaviours or customs (Henley 2020, 72). Since 2012² documenting the world that surrounds us in 360° has become technologically and financially viable, with affordable 360° cameras and devices that can reproduce the acquired footage such as mobile phones and stand-alone head-mounted-displays (HMD). This first generation of 360° cinema allows for three degrees of freedom (3DOF), namely yaw, roll and pitch. Thus, viewer-users can direct their gaze from a fixed point of view (where the camera was set up) towards any point within a spherical video.

Common stylistic elements for driving early VRNF narratives have been testimonials, voice-over narration and witnessing events as they unfold, closely adhering to an expository mode of documentary film as defined by Bill Nichols (2017). In fact, many early 360° VRNF projects focus on news-worthy journalistic subject matter³ driven by news outlets such as The New York Times, The Guardian, The Economist and Vice amongst others. We have observed that the camerawork and editing often follow common stylistic approaches of journalistic nonfiction, with hand-held 360° camera work, privileged perspectives and fast-paced evidentiary editing. Yet, this style easily causes confusion, disorientation and risks motion sickness among viewers of 360° films (Dooley 2021, 33). Therefore, a more slow-paced, fly-on-the-wall style observational mode in 360° VRNF may provide a less mediated approach, which, similar to a traditional 2D observational mode, can be characterized by its use of an unprivileged camera style, long takes and with reduced imposition of editing-induced re-signification (MacDougall 1998). An observational mode of VRNF may allow the viewer-user more freedom to appreciate and explore their surroundings and connect more intimately with content of their choosing.

Two leading aspects that set 360° VRNF apart from traditional 2D documentary films are a sense of presence and the agency of the viewer; affordances that are inherent to this novel technology and thus shift the functioning of a documentary from intact observation to a more active participatory comprehension. Jamie McRoberts (2018, 102) states that ‘[w]ith reference to the medium’s immersive and interactive qualities, this sense of presence propels users closer to the diegetic world, and thus its characters and the conflicts they face’. The sense of immersion achieved by agency in 360° VRNF has been frequently compared to a ‘feeling of being there’ – an attribute associated with the observational mode of Direct Cinema (Leacock 2011; Green et al. 2020). However, in VR this ‘feeling of being there’ may better be described as ‘mediated presence’ through which the user is brought ‘closer’ to the (past or present) real-world experiences of the characters and their presented conflicts. McRoberts (2018, 104) considers the sense of (mediated) presence a ‘multi-faceted concept, initiated through a complex interplay between the technology, the VR content and the user’.

For Janet H. Murray (2016) agency ‘is the single most important design value in any digital artifact’. She associates agency with purposeful actions that meaningfully alter a world (Murray 1997, 128). Therefore, it is the intentionality of the action within a digitally created world that we denote as agency, through which the passive viewer becomes an active viewer-user. Eric R. Williams, Love, and Love (2021, 26) define three distinct types of agency in VR cinema: directional agency (choosing where to look in the story), emotional agency (choosing who to connect with during the story, how those connections should make them feel, and when those feelings should occur) and intellectual agency (choosing what to pay attention to during the story). As the gaming industry is constantly pushing the development of VR, nonfiction contents are evolving, too. Creators are increasingly experimenting with game design strategies, such as controlled interactivity and open-world designs, and implementing six degrees of freedom (6DOF), adding three more possible movements to 3DOF, namely forward/backward (surge), up/down (heave) and left/right (sway). We suggest that such augmentations contribute to the eliciting of two more levels of agency, namely spatial agency (choosing where to move in the virtual world) and interactive agency (choosing how to interact with, and consequently how to alter, the virtual world).

VRNF with 6DOF further challenges previous assumptions and constraints that define documentary film as a recording and representation of a pre-filmic reality at a certain time and place of the past. The perceived temporality of viewer-users may shift, acknowledging that the immersive images presented to them are not mere representations of a temporal past, but more like components of a present-tense experience in which their interaction with the diegetic world is a pre-requisite for narrative progression. Therefore, as VRNF is moving towards open-world interactive designs, fundamental questions arise as to which extent we can still speak of VRNF as ‘documentary’? If we develop 3D environments with interactive elements, putting the user’s agency central to the exploration of virtual worlds, are we still engaged in ‘documenting’ the world that surrounds us? How, then, might an autopoietic mode of VRNF push traditional boundaries of documentary film?

Autopoiesis and creative systems

The term autopoiesis was initially coined by Humberto Maturana and Francisco Varela in 1972 to describe self-maintaining chemical systems of living cells. They define these self-maintaining living systems as.

(…) a system whose organization defines a domain of interactions in which it can act with relevance to the maintenance of itself, and the process of cognition is the actual (inductive) acting or behaving in this domain. Living systems are cognitive systems, and living as a process is a process of cognition. This statement is valid for all organisms, with and without a nervous system (Maturana and Varela 1980, 13) (italics in original).

Autopoiesis is, according to Maturana and Varela, what defines life. On a cellular level, this refers to a heterostatic system consisting of individual parts that reacts to and integrates external stimuli which necessarily requires a level of randomness to ensure the survival of any living system (Maturana and Varela 1980). Due to the openness of Maturana and Varela’s coinage of the term autopoiesis, fields outside biology have appropriated and adapted the concept. In the preface to Autopoiesis and Cognition (Maturana and Varela 1980, 71), Stafford Beer suggests that ‘[t]he fact is that if a social institution is autopoietic (and many seem to answer to the proper criteria) then, on the author’s own showing, it is necessarily alive’. Thus, this openness of the interpretation of autopoietic theory outside of the original – biological – definition has found fertile ground in different disciplines including sociology (e.g. Luhmann 1995), literature (Livingston 2016), and human creativity in general (Gornev 1997), amongst others. Yet, Randall Whitaker (1997, 2) warns that the indiscriminate use of autopoietic theory, due to a lack profound understanding of the body of work published by Maturana and Varela, has led to ‘invocations of the theory which are (e.g.) fragmentary, distorted, or clearly erroneous’.

Despite the critique, the seductiveness of applying autopoiesis outside of biology has been particularly relevant for systems theory. Within the arts, Takashi Iba (2010) uses Maturana and Varela’s autopoietic cognitive systems and Luhmann’s theory of autopoietic social systems to propose a ‘Creative Systems Theory’ that aims at understanding ‘the nature of creation’. He defines a creative system as ‘(…) an autopoietic system whose element is “discovery”, which emerges only when a synthesis of three selections has occurred: “idea”, “association”, and “consequence”.’ (Iba 2010, 6610). In contrast, J. Mark Bishop and Muhammad Majid Al-Rifae point out that Iba’s work remains conceptual and avoids taking the individual artist’s ‘psychic level’ into account. Therefore, they propose that a creative system is ‘(…) a clearly delineated and identifiable network of continuously operational component producing processes and concomitant elements, bounded as an autonomous entity within its own artistic environment’ (Bishop and Al-Rifaie 2016, 1) (italics in original). On the artist’s level they consider ‘(…) the working autopoietic artist as entailing a reduction in complexity, ravenously consuming ‘meaning – distinctions’ within her environment; in this way the autopoietic artist iteratively decodes her environment by continuously first selecting, then processing, areas of meaning’ (Bishop and Al-Rifaie 2016, 5).

Within this self-contained creative system proposed by Bishop and Al-Rifaie (2016), we suggest that a new meta⁴-creative system may arise, in which the viewer-user takes on the role of co-creator or artist (in the widest sense) of decoding virtual stimuli, selecting areas of meaning within the mediated diegesis and processing these through layered levels of agency. In this sense, we consider that agency is a pre-requisite for autopoiesis as it necessitates an intentional action to influence or alter the storyworld in a meaningful way.

Building on these theoretical foundations, in the context of VRNF we understand autopoiesis as the viewer-user’s process within the constraints of a pre-determined virtual environment in which the ‘self’ (auto) engages with virtual and external parameters to create (poiesis) a deeply personalized narrative. The viewer-users’ process is thus twofold. Firstly, they react to the stimuli of a virtual environment, and secondly, they exert influence on the underlying programming that in turn may change certain aspects of the actually experienced virtual environment. Virtual parameters determine how the viewer-user can move in and interact with the virtual environment, as well as limiting the boundaries for exploration. External parameters that influence the viewer-user’s process are varied and can be mediated (as part of an entire experience) or unmediated (random occurrences during the experience). These include the required hardware set-up, the physical properties of the space surrounding the viewer-user’s experience, and environmental factors, such as temperature, humidity, noise, etc. For example, in Alejandro Iñárritu’s mixed reality installation Carne y Arena (2017) about migrants attempting to cross the US-Mexico border, viewer-users are led through mediated physical spaces both before and after the multi-sensory VR component of the experience. Viewer-users are thus primed to interact with the VR experience in a certain way based on external factors (the way the physical space is set up), react to the stimuli provided by the VR system and, through agency, alter and change the experience to which the underlying programming reacts (i.e. continuously and seamlessly changing vantage points, connecting with different characters within the diegetic world, etc.).

Consequently, even though not all documentaries restrict their experiences to revisiting past events, within VRNF this may lead to a clear dissociation between the perceived past through presenting fixed 360° video footage, and the experience of agency-induced autopoiesis as unfolding during the actual viewing. Thus, autopoiesis occurs to a lesser degree in VRNF with 3DOF and to a greater degree in VRNF with 6DOF.

Autopoiesis through agency in expository VRNF

Early 360° VRNF experiences adhere closely to traditional observational and expository documentary modes or a combination of the two (Nichols 2017; Green et al. 2020), utilizing the breadth of stylistic approaches that are common in their 2D counterparts.

The expository mode of 360° VRNF aims to contextualize the viewer-user through the use of voice-over, which can be in a ‘voice-of-god’ style, a ‘knowledgeable expert’, or reflexive in nature, potentially overlaying intertitles and applying evidentiary editing.⁵ Clouds over Sidra (Arora and Milk 2015) and The Fight for Falluja (Solomon 2016) both utilize primarily expository modes.

Clouds over Sidra

Clouds over Sidra is about a 12-year-old girl living in the Za’atari refugee camp in Jordan. A linearly structured reflexive voice over, dubbed by Sidra, helps viewer-users orient themselves in spatial terms and then interpret what they (are supposed to) see. The non-diegetic music track that accompanies the scenes works rather traditionally, aiming to trigger an emotional response from the immersed viewer-user. According to creator Chris Milk, Clouds Over Sidra has an unparalleled empathetic effect due to the sense of presence that can be achieved by 360° video (Milk 2015). Yet, his now almost infamous claim that VR is the ‘ultimate empathy machine’ has been widely challenged (Murray 2016; Fjellhøy 2017; Nash 2018b; Eitzen 2020) Figure 1.

Figure 1. Watch Clouds over Sidra.

The fight for Falluja

The Fight for Falluja (Solomon 2016), commissioned by The New York Times, also employs a linear narrative, voice-over and evidentiary editing. War reporter Ben Solomon is embedded in Iraqi military operations attempting to re-capture the city of Fallujah from ISIS. His voice-over directs the user’s attention to intendedly relevant visual information and provides an interpretation to help understand what can be seen. Sporadic testimonials dubbed into English by Fallujah’s inhabitants give viewer-users brief glimpses into their lives. A non-diegetic music track punctuates emotive moments and adds to a tense atmosphere. On-screen text overlays divide the VR experience into two parts to help orient the viewer in time (The Battle and The Aftermath) Figure 2.

Figure 2. Watch The Fight for Falluja.

Both experiences rather straightforwardly apply traditional 2D documentary narrative structures to a 360° environment. The voice-overs attempt to direct the viewer-users’ gaze, as well as how they connect emotionally and intellectually with the content. By doing so, both cases are continuously working against the just-earned freedom of choice and thus limiting viewer-users’ sense of presence and agency. Our simple observation is in line with the conclusion of the study by Miguel Barreda-Ángeles, Aleix-Guillaume, and Pereda-Baños (2020), who have suggested that the cognitive resources needed to achieve a sense of presence during VR immersion reduces the ability to process the story. Thus, purposely or not, 360° VR creators often counteract the enhanced affordances of the VR platform just to maintain their narrative’s intelligibility. Indeed, the viewer’s attention is split between exploring the environment and following the narration: non-diegetic components such as text overlays and voice-overs limit the sense of presence, leading Murray (2016) to call on VR creators to leave out anything that is ‘not diegetically part of the virtual space’.

Autopoiesis through agency in observational VRNF

In contrast to the expository mode of VRNF, with an observational mode the viewer-user takes on the role of a non-participant observer, in control of the framing, with full autonomy to decide which protagonists to connect with and which (sub-) story to follow. Narrative mediation is reduced, allowing viewer-users to actively explore their environment and freely make sense of the storyworld.

Nomads

An example for observational VRNF is the three-part series Nomads (Lajeunesse and Raphaël 2015). Filmed in Kenya (Maasai), Mongolia (Herders) and Borneo (Sea Gypsies), the viewer-user finds themself in the midst of the lives of three nomadic communities. The long-take scenes show their customs and religious rituals, such as a herder’s family preparing and drinking yak’s milk in a yurt, children paddling a dug-out canoe from one stilt-house to another, and a Maasai fertility dance. There is no use of voice-over, intertitles, or sub-titles that may provide direction, context or translation, leaving viewer-users to try to discover and make sense of their surroundings on their own. Creator Félix Lajeunesse knowingly comments that this experience was intended to ‘push the boundaries of virtual reality as a medium to create intensely personal experiences’ (Hayden 2016). Following such intention of non-intendedness, viewer-users become co-creators of and co-subjects within these virtual worlds, choosing where to direct their gaze, who and how to connect to within the represented environment. Thus, in the closed creative system of a VR application, autopoiesis occurs by processing meta-environmental stimuli that, due to a variety of internal (virtual) and external factors, leads viewer-users to intentionally create a multiplicity of personal narratives with every ‘visit’. In both expository and observational modes of VRNF no two viewings will be the same due to the infinite possibilities of personalized framing options within 360° cinema. However, in the observational mode of VRNF the inter-subject correlation between viewing behaviour is potentially lower than in the case of the more directed experience of the expository mode of VRNF in which the chance for similarities in the actualized narratives are relatively higher Figure 3.

Figure 3. Watch Sea Gypsies Intro.

Challenges of observational VRNF

As with traditional 2D observational cinema, observational VRNF faces similar ethical challenges. According to Kate Nash (2018a, 125),

VR runs the risk of producing improper distance and an ironic mode of moral engagement when it invites forms of self-focus and self-projection rather than a more distanced position that allows for recognition of distance between the self and other.

To some extent, Nomads risks producing this improper distance and may lead to romanticizing and exoticizing the ethnographic content it displays, becoming a ‘cinema of attractions’. Instead of bridging cultural divides, the lack of contextualization may further alienate viewer-users with culture-centrist biases. Furthermore, in the attempt to fully immerse the viewer-user, any reflexive reference to the filmmaking process is painstakingly avoided, creating what in 2D documentary film would be considered an anachronistic fly-on-the-wall illusion. In the quest for ever-greater immersion this avoidance of reflexivity drives the style of observational VRNF. As Lajeunesse and his co-creator Lajeunesse and Raphaël (2015) state in their director’s commentary of Nomads: Sea Gypsies: ‘We had to solve problems, you know, how to hide a 20-person crew when shooting 360 on the open sea’.

Observational 360° VRNF may indeed enhance the viewer-user’s sense of presence, yet their agency is still limited to directional, emotional and intellectual means. The viewer-user remains ghost-like, that is, being part of but invisible within the entered diegesis, unable to interact with the represented world. Thus, the prevalence of what Mandy Rose terms ‘agency of perspective’ (or directional agency) can create a voyeuristic or surveillance effect (Rose 2018). This sense of presence, being bodiless and invisible to the diegetic world, has been described as the ‘Swayze Effect’ (Burdette 2015), in reference to the film Ghost (Zucker 1990).

With the exception of still-experimental volumetric video capture,⁶ 360° filming techniques can only provide experiences limited to 3DOF. Here, the viewer-user’s agency is to a great extent non-directed thus unintentional and limited to directional, emotional and intellectual agency. Considering this, autopoiesis may occur when the viewer-user intentionally re-directs their gaze, thus they will have to make a concerted effort to choose alternative points of focus within 3DOF VRNF. However, in both expository and observational VRNF, the viewer-users’ ability to meaningfully alter the virtual environment or their role within it is limited. If, on the other hand, viewer-users are afforded spatial and interactive agency in VRNF experiences with 6DOF, intentionality becomes central to the viewer-user’s agency and with it, the possibility to alter the virtual environment and narrative outcomes, thereby enhancing the autopoietic potential of the VR system.

Autopoiesis through agency in VRNF experiences with 6DOF

David Philip Green et al. (2020, 4) distinguish between linear VR and interactive VR, with the former being mainly limited to 360° video and the latter involving forms of interaction that require additional hardware. Aimed at understanding the interactions of a non-specialist audience with VRNF, the authors mention that the high-resolution computer-generated home screen of the (now obsolete) Oculus Go HMD, as well as other explorable virtual environments, proved to be a popular refuge from the often-challenging 360° VRNF experiences: ‘(…) this expressed a pleasure in immersion in high-resolution images, and in an environment in which they had the freedom, agency and time to explore as they chose’ (Green et al. 2020, 21). Many second-generation consumer HMDs are capable of 6DOF, in which users can move within a virtual environment (surge, heave and sway), and allow for manipulating objects with the help of controllers or hand-tracking technology. The ability to influence virtual environments through intentional spatial displacement and interaction gives users new and enhanced levels of agency in VR experiences.

When introducing spatial and interactive agency to VRNF, VR filmmakers now have a new set of tools and can truly move away from traditional linear storytelling techniques of 2D nonfiction. This allows for autopoietic VRNF experiences in which viewer-users can create their own ever-evolving storylines. While i-docs explore novel ways of engaging with filmed material, they still, in general, rely on edited sequences that can be played in different orders on a flat screen to create unique user-defined experiences. Within VRNF, the boundaries between the screen and the viewer-user dissolve and viewer-users are more inclined to feel part of the experience. They become integral to the autopoietic system. Viewer-users must make intentional choices and carry out actions that shape the unfolding narrative and may even alter the virtual environment. They have to decide when and with which objects or characters to interact, or where to position themselves within a navigable virtual 3D world they actualize through their subjective engagement.

Spatial and interactive agency requires VRNF creators to adopt methodologies of volumetric image capture that go beyond mere monoscopic or stereoscopic 360° cinematography. In order to ‘step in’ and then relocate freely in a virtual space, the environment’s geography and its interactable objects or protagonists need to be digitally recreated. Currently, this can be achieved either by 3D computer modelling that renders whole new fictional worlds, or via photogrammetry, a technique with which volumetric point clouds can be interpolated and rendered photo-realistically from a large quantity of photographs taken of real-world places, objects or people. The question remains, if these recreated worlds that allow for 6DOF still represent a pre-filmic reality and, as such, whether the resulting experiences fit traditional definitions of what documentary is. As if caught in the same dilemma, the creators of the following examples avoid labelling their work as being nonfiction or documentarian. Instead, they call their VR applications ‘experiences’. Due to 6DOF VRNF’s higher degree of participatory and actualizing freedom, a clear labelling may therefore be, quite understandably so, elusive.

Home after war

The VR experience Home After War (Parameswaran 2018) combines photogrammetry with 360° videos. Set in a damaged house in Fallujah, Iraq, Ahmaied, the patriarch of a family tells the audience how he lost his two sons to an Improvised Explosive Device in their boobytrapped home. By using photogrammetry, Home After War allows users to experience full spatial agency within the perimeters of the photorealistic 3D model of the house. The explorative experience is interrupted by the appearance of Ahmaied telling his story. Trigger-zones initiate or pause Ahmaied’s interview and launch 360° videos of daily life in Fallujah. With simple point-and-click interaction, the experience takes a step towards interactive agency. Users have some freedom in choosing when or if they view Ahmaied’s interview or the 360° cut scenes, thus contributing to an autopoietic experience through personalized temporal ordering (order of selecting the cut scenes) and spatial ordering (order of exploration) of the representational space and inducing a co-creative realization of the experience’s potential narrative Figure 4.

Figure 4. Watch Home After War Trailer.

Through its (still limited) use of interactive agency Home After War approaches a game-oriented design of VRNF and therefore provides greater autopoietic potential compared to 3DOF VRNF. Note that with this the boundary between video games and game-like VRNF becomes increasingly blurry. The following two VRNF experiences actively include game-oriented elements that fit snugly into our concept of autopoietic VRNF, yet further challenge the notion of ‘non-fiction’.

The key

Celine Tricart’s VR experience The Key (2019) makes ample use of spatial and interactive agency. Exposed to a series of dream-like computer-generated scenarios, viewer-users are presented with simple tasks and decisions that shape the narrative. Through visual metaphors, such as changing colour palettes to elicit subtle emotional reactions, multi-eyed monster-like shadows standing in for a dystopian bureaucracy, or helpful mermaids, Tricart intends to stimulate the feelings of loss, fear, helplessness and hope that migrants go through when fleeing from war or poverty. The voice-over of an undefined migrant woman recruits the viewer-user to help reclaim her lost memories. She asks them to save three ‘companions’, balls of colour and light with their individual ‘personalities’. These are inevitably lost one-by-one during a journey through barren watercolor-style landscapes. The viewer-user must decide which companion they will sacrifice – a painful choice that, according to the voice-over narrator, ‘has consequences’. In The Key, the painted CGI style eventually gives way to a volumetric photogrammetry model of a bombed-out house in a war-torn city. This ending is reminiscent of the final scene of the animated documentary Waltz with Bashir by Ari Folman (Folman 2008). After 90 min of animation, Folman inserts news-reel footage of the Sabra and Shatila massacre to show that real people were killed, putting the film into perspective by verifying its claims (Rosenkrantz 2011). Analogous to Waltz with Bashir, the final photorealistic scene in The Key serves as a stark reminder that migration is not a game, educing a sense of real loss. Here, the voiceover explains what some of the metaphors in the dream-like scenes mean and presents the viewer-user with startling facts on the dire situation of migrants worldwide Figure 5.

Figure 5. Watch The Key Walkthrough.

By enlisting the viewer-user to actively contribute and shape the unfolding narrative, Tricart generates an autopoietic system that operates on agency within a reactive virtual environment.

Goliath

The VR experience Goliath: Playing with Reality (Murphy and Abdalla 2021) retraces the homonymous protagonist’s struggles with schizophrenia. This experience intends to situate the viewer-user within Goliath’s state-of-mind and attempts to question their own sense of what is real. At the start of the experience, Tilda Swinton’s voice-over reminds us that ‘Reality is a story we tell us to make sense of the world. Some stories aren’t true. This one is’. She continues, ‘Everything you see is just part of the game. Everything you feel is real.’ In doing so the voice-over sets the cognitive frame that Goliath’s unfolding story and his testimonies really are truthful. By explicitly indicating that the viewer-user’s feelings within the VR experience are ‘real’, the VRNF experience stimulates a reflection upon how the ‘self’ relates to the storyworld as an integral part of the autopoietic system. The viewer-user must intentionally influence and become part of the diegetic world by, for example, controlling an in-world 8-bit style video arcade game or by assuming the role of an avatar in an elaborate first-person shooter. In the first section of the experience, the viewer-user is asked to record their name. Towards the end of Goliath: Playing with Reality, this voice recording is played back to the viewer-user, distorted by echoes. By actively fusing something as personal as one’s own voice with the diegetic world, Goliath: Playing with Reality dissolves the boundaries between videogame, VRNF experience and the subjectivity of the viewer-user’s role in the unfolding narrative. Viewer-users actively change the virtual environment through their intentional actions, i.e. they can shape virtual elements with the HMD’s controllers and, in an ultimate uncanny moment, their voice fuses with the VR system, becoming an integral part of the narrative and consequently the autopoietic system Figure 6.

Figure 6. Watch Goliath Trailer.

‘Reading’ virtual reality as non-fiction

These 6DOF VRNF experiences go beyond the mere audiovisual reproduction of a filmed real event from the past. The question therefore remains if they, particularly the two latter cases, can still be considered as nonfiction, ambitioning some inherent ‘truth claim’, or should such ‘experiences’ be instead seen as some non-fiction video games based on an achievement-reward system. A key differentiation may therefore lie in how 6DOF VRNF with spatial and interactive agency may be defined within a nonfiction cognitive frame.

A core assumption of ‘reading’ a film as documentary has been proposed by Noël Carroll (2003, 169): ‘The film and the writing come labelled, or, as I say, indexed, one way or another, ahead of time.’ (italics in original). In consequence, documentary films are labelled, or indexed, as such, setting a cognitive frame for viewers to read the film as a representation of reality. Ros et al. (2018, 229) suggest that documentary could therefore be defined.

as a cognitive frame, a specific set of expectations that can be applied to film on the basis of its adherence to culturally established formal conventions (which may change as our visual culture develops) and the ways in which it is framed socially and contextually. (which may also change as documentary constantly acquires new social functions)

Consequently, the ‘mode of reception or cognitive frame’ of documentary evolves dynamically and emerging techniques for creating VRNF, such as 3D modelling, user-centered or game-oriented designs may be considered reflective of creative techniques used in traditional 2D documentaries such as re-enactment, animation and montage (Ros et al. 2018, 239). The viewer-user’s awareness of a cognitive frame (their recognition of an intentional representation of reality by the makers) might differently shape the experience, that is the different affordances of agency between ‘traditional’ documentary and VRNF. Thus, ‘reading’ a film or VR experience as nonfiction may happen externally, for example by appropriately labelling a VRNF experience in its publicity materials indicating its genre, or internally, that is, within the VRNF’s created virtual environment, through audiovisual triggers, such as camera style, use of interviews, voice over narration, etc., that viewers are accustomed to when watching a traditional linear documentary. This is certainly true for VRNF experiences with 3DOF, in which the appropriate label and use of an observational or expository mode suffice for the user to identify the experience as a representation of real-life events. However, as we move towards interactive 6DOF VRNF experiences, particularly when they are not always labelled as such from the outset, generic thresholds between VR documentaries and VR games become increasingly blurry, as one could ponder upon the generic categorization of Tricart’s game-like VR experience The Key or Murphy and Abdalla’s Goliath: Playing with Reality. No wonder then that the makers simply labelled their work as ‘VR experience’. Yet, despite relying heavily on artificially modelled 3D worlds, we may still find indexations, or cognitive triggers, that indicate a certain realist intention and that differentiate them from ‘mere’ video games. These can be auditive in nature, for example embedded within the voice-over narration, testimonials or soundscape, or visually in nature, for example through the integration of real-life photographs, documentary style video footage (2D or 360°) or via volumetric photogrammetry techniques that can recreate real environments with a high degree of photorealism.

With the advancement of volumetric video capture technologies, and more readily available and precise photogrammetry, we might speculate that future VRNF will evolve further towards non-linear open-world autopoietic experiences that are temporally present, requiring the viewer-user’s agency to exist, and are therefore non-repeatable and highly personal.

Conclusions

VRNF, with 3DOF or 6DOF, differs fundamentally from linear 2D documentary in that users are afforded non-linear narrative exploration, increased degrees of active agency and a sense of embodiment. Building on traditional 2D documentary approaches, VRNF with 3DOF can be characterized by the dominance of a mediated diegesis, in which viewer-users may direct their gaze to any content area from a fixed point-of-view within a ‘frameless’ image, but are guided by visual, auditive or graphical cues that follow a pre-determined unidirectional storyline. Viewer-users can intentionally or unintentionally direct their gaze (directional agency), who and what to connect with emotionally (emotional agency) and intellectually (intellectual agency), thus creating a personalized ephemeral experience that exists only once and for the duration of their participation in it. In VRNF with 6DOF viewer-users are afforded two more levels of agency, namely spatial agency and interactive agency. These added levels of agency empower viewer-users to intentionally shape the storyworld and potentially alter the virtual environment in an experience that grows organically through the multiplicity of potential narrative outcomes. Whereas interactive documentaries may attain certain levels of autopoiesis, it is the enhanced agency and embodiment afforded by VRNF that engages viewer-users in an autopoietic system as they decode and reduce the complexities of the diegetic world through directional, emotional and intellectual agency to then process these areas of meaning through spatial and interactive agency. Thus, the viewer-user’s agency, prompted by internal (virtual) and external (real-world) factors, while introducing a degree of randomness within the closed system of a VRNF application, leads to a self-perpetuating system that can only exist within the complex interactions between the (absent) original creator, the technological system (the VRNF experience with its hard- and software components) and the (present) viewer-user.

Auditive or visual indexations provide the viewer-user with cognitive triggers that signal the relation to reality of the VRNF experience, with the requirement of at least one reference to ‘real people’, ‘real events’ or ‘real places’. The sense of presence, or ‘being there’, can create the illusion of witnessing people, places and unfolding events, as if they were actually happening to the viewer-user. Whereas in traditional 2D documentary filmmaking there is an acceptance that the filmmakers ‘witnessed’ and recorded a place or an event in a pre-defined moment in time, the sense of presence and required levels of agency within VRNF changes the temporality of the viewer-user (see also Nash [2018b]). In 3DOF VRNF they can now witness a meta-environment first-hand, accepting that they exist within a pre-defined spatial representation of the temporal past while simultaneously being an actualizing part of it in the temporal present. With 6DOF VRNF the realization of a temporal past dissolves, as a concerted effort at exercising agency is required to shape both the narrative and the virtual environment. Thus, any VRNF has an added dimension of actuality and presence as the viewer-user’s intentional actions are required to create the storyworld, making them an indispensable component for the autopoietic system.

An autopoietic mode of VRNF can thus be defined as a self-maintaining closed creative system that relates directly or indirectly to real events, real people and/or real places, within which the viewer-user is afforded sufficient levels of agency to decode and process areas of meaning and engage within the diegesis of a delineated virtual environment in potentially limitless ways. This assumption may therefore pave the way for future development of VRNF content that has the potential to create unique and unrepeatable nonfiction content in the present tense with unlimited narrative outcomes. Furthermore, we may speculate that VRNF could eventually include multi-user elements similar to VR chatrooms that not only allow for the simultaneous immersion of viewer-users but would actively necessitate the protagonists’ agency within a co-creative, autopoietic virtual space.

Finally, and ultimately, autopoietic VRNF seems to steer away from a formalist criterion-bound definition of documentary (resulting in a ‘product’) towards a less tangible understanding of a more phenomenological and subjective experience (resulting in an ‘event’). By introducing an autopoietic mode to VRNF, we need to re-consider previously held assumptions on documentary film as a representation of a pre-filmic reality and move towards perceiving the autopoietic viewer-user as an ephemeral co-creator in conjunction with a reactive VR system as equally valid and deserving of the term ‘non-fiction’, or ‘documentary’. The shift occurs when the viewer-user’s cognitive experience ultimately leads to an acceptance that their engagement with the VR system is a real experience notwithstanding its virtual nature. For, to repeat Tilda Swinton’s suggestive conclusion in Goliath: Playing with Reality (2022), ‘Everything you see is just part of the game. Everything you feel is real’.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Universidad del Magdalena: [grant number Programme for Further Education in Teaching and Research Grant, Academic Accord No. 17, 2019].

Notes on contributors

Andrew Simon Tucker

Andrew Simon Tucker is a documentary filmmaker and associate professor of documentary film at the Film and Audiovisuals Department, University of Magdalena, Colombia. He holds a Master of Arts degree in Visual Anthropology from the University of Manchester, U.K., and is currently pursuing his Ph.D. at the Department for Arts, Culture and Media, University of Groningen, The Netherlands. His documentary films have been screened at film festivals around the world, television channels and streaming sites. His interests lie in collaborative documentary filmmaking and expanded reality nonfiction.

Miklós Kiss

Miklós Kiss is associate professor of Audiovisual Arts and Cognition at the Department of Arts, Culture and Media of the University of Groningen, The Netherlands. His research focuses on contemporary audiovisual media through intersecting narrative and cognitive approaches. His recent books include a work about videographic criticism (Film Studies in Motion: From Audiovisual Essay to Academic Research Video, with Thomas van den Berg, Scalar e-publication, 2016), and about contemporary complex cinema (Impossible Puzzle Films: A Cognitive Approach to Contemporary Complex Cinema, with Steven Willemsen, Edinburgh UP, 2017).

Notes

1 From Greek: αὐτo- (auto-) ‘self’, and ποίησις (poiesis) ‘creation, production’.

2 See also A History of Virtual Reality Nonfiction 2012–2018 (University of Bristol 2022): https://dl.acm.org/doi/10.1145/3290605.3300736.

3 See, for instance, Vice News VR: Millions March NYC (Milk and Jonze 2015), Hong Kong Unrests (Miller 2014), RecoVR Mosul – A Collective Reconstruction (Chen and Schneider 2015), amongst others.

4 The terms ‘meta’ and ‘metaverse’ have, in recent years, been over-utilized for a hyped-up commercialization of VR products and services. Here we use the term based on the original Greek prefix μϵτά- (‘besides, beyond’) to describe something being about what this something is.

5 Evidentiary editing refers to the use of illustrative images that reinforce, evoke, enhance or contradict the content of interviews, testimonials or voice-over narration.

6 High-tech companies have been experimenting with volumetric video capture technologies in which creators and viewer-users may move freely around a scene, such as Microsoft’s Metastage (2018), Intel Studios (2022) or Sony’s Double Dutch (2020). Google’s Immersive Light Field Video (Broxton et al. 2020) will provide a mobile solution that permits (limited) 6DOF.