The immersive archive

ABSTRACT

During the Covid-19 pandemic restrictions of 2020–2021, Melbourne’s Malthouse Theatre created an immersive theatre performance called ‘Because the Night’, comprising six unique storylines within intricately designed spaces. Unfortunately, the production was interrupted and eventually curtailed prematurely due to further lockdown restrictions closing theatres to the public in 2021. The Malthouse Theatre collaborated with Deakin Motion Lab to document and create an archive of the production. This paper considers the team's approach to creating an immersive archive using cutting-edge 360-degree filming technologies that provide a resource for understanding the particularities of immersive performance praxis. As the audience's embodied and emergent experience of Because the Night was essential to the work’s dramatic form as an immersive theatre production, simulating these qualities within the resulting platform was critical. The resulting archive contributes to the ongoing efforts to create new embodied research methods that allow for a wider audience to understand and experience the unique characteristics of immersive theatre using emerging immersive digital technologies.

KEYWORDS:

• Multimedia archives
• documenting performance
• immersive theatre
• immersive technologies
• 3D visualisation technologies

Introduction: the problem of documenting immersive theatre

The 2020 Covid-19 pandemic presented Melbourne’s Malthouse Theatre with an opportunity to innovate new ways of collaborating as a company and to present and preserve their work amidst a crisis. With audience numbers limited and work-from-home restrictions in place in 2021, the Malthouse saw these constraints as a unique opportunity to pursue a bold project that would have been challenging with business as usual. Directed by Matthew Lutton, the entire Malthouse team worked together collaboratively and remotely to create Because the Night, an original immersive theatre production inspired by Shakespeare's Hamlet and produced on a scale never seen before in Australia. The work consisted of six independent storylines following each of the play’s main characters through a maze-like configuration of thirty-five elaborately dressed rooms as a version of the play’s setting of Elsinore. The audience could explore and choose their own emergent trajectory through this story world and its unfolding narrative.

Unfortunately, the production was interrupted and eventually curtailed prematurely due to further lockdown restrictions closing theatres to the public in 2021. As the production involved an enormous effort resulting in technical innovations in performance production and an unprecedented way of collaborating across the Malthouse team, there was an urgent interest in documenting the performance. The Malthouse Theatre collaborated with Deakin Motion Lab to document and create an archive of the production before the world of Elsinore was dismantled.

To achieve this, the Development Team at Deakin Motion Lab created 'Because the Night Interactive Archive', an alternative digital experience and immersive archive of the performance within a game engine platform (Greuter et. al. 2022). Additionally, the Documentary Team conducted video interviews with the entire production team and cast about their experiences, methods and processes in devising, producing and performing the production. This paper's co-authors include members of both project teams: Greuter and Mulvany (Development and Documentary Teams, with expertise in game design) and Myers (Documentary Team, with expertise in immersive and participatory performance, performance documentation and digital archives).

This paper considers the team's approach to creating an immersive archive using cutting-edge 360-degree filming technologies that provide a resource for understanding the particularities of immersive performance praxis. The audience's embodied and emergent experience of Because the Night was essential to the work's dramatic form as an immersive theatre production. Therefore, simulating these qualities within the resulting platform was critical in creating an archive of the project.

While a systematic survey of the current methods of archiving and re-producing performance is beyond the paper's scope, the paper considers issues in the field of performance documentation and digital preservation, along with recent projects exploring research methodologies for capturing immersion, ephemerality and performative characteristics unique to immersive theatre. The prototype developed provides an appropriate resource for studying the theatrical praxis of the complex genre of contemporary immersive performance. It also offers an alternative venue for experiencing immersive performance within a computer environment.

The immersive world of Because the Night

From March to August of 2021, Because the Night transported audiences to the fictional 1980s logging town of Elsinore, where themes of corruption, love and revenge intertwined amidst the backdrop of a workers’ rights movement and the sudden death of the king. The town's inhabitants were shrouded in whispers and eruptions, as tensions escalated and secrets concealed within the ancient forest threatened to bring about chaos during the town's annual winter solstice carnival.

In this adaptation of Shakespeare’s Hamlet, old Hamlet was killed by his sibling, re-gendered as Claudia, who was engaged in a highly-charged relationship with his son Prince Hamlet's mother, Gertrude. Instead of focusing on Prince Hamlet’s revenge plot or delving into his deep existential horrors, Because the Night presented the play as an experiential production that focused on the multiple perspectives and possible storylines of each character.

The world of Elsinore was comprised of thirty-five cinematographically dressed spaces in an 1800 sqm area, inhabited by six protagonists who performed across these spaces. As audience members entered the theatre they were given masks and cloaks to wear and were instructed not to use phones or talk during the performance. This is what Janet Murray refers to as the ‘threshold marker’ that structures participation in immersive narrative experiences, signals the crossing of the boundary into an immersive reality and where the audience member leaves parts of themselves behind (Murray 1997, 113). Audiences were instructed to freely roam about the set, examine props to gather information, follow any of the six performers along their independent and intersecting storylines, piece together their stories and uncover hidden mysteries. This behaviour aligns with procedural thinking with computation, the ‘codified rendering of responsive behaviours’ (Murray 1997, 74), which contributes to computer’s abilities to create environments that are both procedural and participatory, which Murray acknowledges is what is usually meant when computers are described as interactive (Murray 1997). With Because the Night, audiences were invited to play the role of silent witness. While there were no explicit invitations for direct interaction or dialogue between the performers and audience members, they were encouraged to interact with the space, to be curious, playful and exploratory and to uncover hidden narrative elements.

Elsinore was meticulously designed, with each room or set filled with hidden details and props to enrich the narrative. Some audience members took it upon themselves to try on costumes found in a wardrobe of one room and wear them throughout the performance. Locked rooms could be accessed if they observed a character using a secret code to enter or if they decoded hidden messages embedded in the scenography to discover the code themselves. Audiences could also interact with the set by playing tape recordings on a cassette player or playing back various types of VHS found in parts of the sets and searching for hidden information on those media (see Figure 1).

Figure 1. Promotional photography of ‘Because the Night’, showing the security room containing logs, dossiers and VHS tapes available for audience interaction

Audiences were given the freedom to wander through the rooms, to follow the performers and examine narrative props, giving them a greater understanding of and complicity within the fictional world and its subplots. As silent witnesses, the audience members’ decisions and interactions influenced which characters and storylines they encountered, the depth of knowledge they gathered about the world and which spaces they experienced. This could be understood as the work’s realisation of a sense of agency, which Murray defines as ‘the satisfying power to take meaningful action and see the results of our decisions and choices’ (Murray 1997, 126). The audience’s engagement with objects, points of view and trajectories chosen through the performance determined how they experienced an overarching narrative. While there was a beginning and an end to the performance, it was non-linear and comprised of multiple storylines and possible perspectives. However, an audience member could choose to gain a single perspective on the events and interactions of the characters by following just one character throughout the performance.

For example, following Prince Hamlet would reveal his journey of revenge and descent into madness as he struggled with his conflicting emotions and tried to uncover the truth about his father’s murder. The audience would witness the murder of Polonius (Counsellor to Claudia and Laerte’s father) and Hamlet's relationship with Ophelia, Claudia’s daughter, which was marked by love and betrayal.

On the other hand, following Claudia would reveal her scheming and manipulative behaviour as she tried to advance her own interests and manipulate events to shift the blame for Old Hamlet's murder onto Hamlet. The audience would witness Claudia poisoning a knife and later passing it to Hamlet for use in a duel provoked by Laertes to avenge his father's death (Polonius).

In reality, however, the audience usually followed several characters, each with different storylines. This meant that the performance was emergent for the audience, but also, to a degree, for the performers as well. Each performer had their own repeatable and rehearsed sets of actions and trajectories through the performance, but the transitions between rooms were coordinated with all the other characters to happen simultaneously. A cueing system was devised as a means to communicate the timing of the transitions between the performers in different rooms. This system involved a combination of lighting and sound effects operated by a technician who observed the performance from a booth set high above the entire performance space. The cueing system was required because the timing of each performer’s actions in a given space could vary depending on the audience’s own emergent and unpredictable movement and interactions within the space. For example, an audience member sometimes picked up a prop that a performer needed for an action to proceed, thereby, delaying the transition. As the trajectories of all the other characters were interconnected with the transitions, the performers had to adapt in the moment to improvise and extend a scene until the cue. This brought an emergent, responsive and changeable quality to the performance.

An immersive archive

Immersive theatre is a genre that seeks to exploit all the experiential elements of performance and place the audience at the heart of the work (Machon 2016, 29). By ‘heart’ Machon implies both the physical and imaginative incorporation of the audience into the world of the work as distinctive of this genre, both in terms of the audiences’ perceiving and feeling bodies and their desires and volition that may lead to various forms of involvement or interaction within the work. The challenge in documenting and archiving an immersive work like Because the Night was to capture the experiential heart of the work.

The immersive work Because the Night is embodied and emergent and its creation is determined by the paths that audience members choose to navigate through the production. As such, Because the Night did not offer a traceable material original event. Its bifurcation of time, multiple storylines and infinite possibilities make it impossible to experience all six storylines within one run of the performance. Even if experienced over another six runs, subtle transformations that were staged in other spaces of the world simultaneously would still be overlooked.

In this way, the immersive performance presents a challenge to traditional ways of conceiving of the relationship between performance and document as ontological, ‘with the event preceding and authorising its documentation’ (Auslander 2006, 1). It already resists the singularity or authenticity privileged by the imperialistic cultural habits of archival logics, which Schneider has argued ‘demand that performance disappear’ in the process of saving it (2001, 102). However, this archival thinking is also an intrinsic property of the particular aesthetic form of immersive theatre realised by Because the Night. This property is what contributes to its expressive power and its immersiveness.

Because the Night achieved a simulation of a holistic world through its elaborate displays of fictional material remains that indicate a world with dramatic richness of past events, a particular form which was pioneered in the work of UK company Punchdrunk. In this form of immersive theatre, the audience is invited to explore haunted spaces dressed with objects, artefacts and documents that provide clues to histories, back stories and traces of memory of preceding actions. This aspect of the performance is related to what Murray describes in Hamlet on the Holodeck as the encyclopaedic capacity of the computer (1997, 89).

The encyclopaedic quality makes the storyworld ‘seem as explorable and extensive as the actual world’ (Auslander 2006, 71). This contributes to Murray’s definition of ‘immersion’ and the immersive experience, as ‘the sensation of being surrounded by a completely other reality’, that is, in a simulated world or alternate reality (Auslander 2006, 98). James Frieze describes immersive experiences as ‘event bubble[s]’ in which the reality of the wider world is excluded to allow an audience to enter into the world of the event (2016, 5). The ability to represent an alternate world with breadth, detail, spatial and temporal depth and dimension, affords the possibility to tell stories with multiple perspectives and storylines, to ‘form a dense and wide-spreading web’ (Murray 1997, 84). Murray argues that this property, along with those of the procedural, participatory and spatial, makes computers the ideal medium to express complex and immersive narratives that reflect contemporary ways of perceiving and experiencing the world (Murray 1997, 89). The same could be argued for immersive theatre and for the research methodologies necessary to enable the documentation of the complex and multiple perspectives within a performance form that puts the audience’s embodied experience at its heart.

A key difference in form between Because the Night and an immersive production such as that of Punchdrunk, is the emphasis on the spoken text. Where narrative is primarily expressed by the performer through dance and movement in Punchdrunk’s work, Shakespeare’s text was performed in Because the Night through dialogue and soliloquies spoken by the performers in proximity to the audience. In formulating her position on the computer’s representational power, Murray begins with the argument that the digital format is better suited to represent some kinds of knowledge than print (Murray 1997, 6). She offers the example of presenting multiple versions of Hamlet’s soliloquies alongside the text, instead of reading the text on its own. With its capacity to present texts, images and moving pictures, she argues the computer offers the precision of reference, greater control over diverse sources and forms of information that enable new directions of inquiry. However, the performance and experience of Because the Night are sites of such inquiry and knowing. It is a ‘performative archive’, which is how Schneider argues performance remains (Schneider 2001, 102). She suggests, ‘the place of residue is arguably flesh in a network of body-to-body transmission of enactment – evidence, across generations, of impact’ (Schneider 2001). It is a collective memory that is accrued and secured through ritual repetition, through acts of reading ‘performative relations to documents and documents’ ritual status as performatives within a culture that privileges objects’ (Schneider 2001, 105).

From Schneider’s argument, it can be understood that the experience of the performance and the traces of it in the document or archive are performative acts and sites of performance. This argument begins to recuperate experience and traces of presence in performance denied by earlier postmodern critiques of media-based performance, notably Phelan’s ontological orientation of performance towards its ‘disappearance’ (1994, 146–166). New debates, such as those explored in Giannachi, Kaye and Shanks’ collected volume Archaeologies of Presence, bring to the fore questions on the temporal or processual dimensions of presence, ‘the tense of presence’ (Giannachi, Kaye and Shanks 2012, 7). This ‘recovery’ is particularly helpful in understanding the ways that digital methodologies can situate and manifest the temporal, embodied, volitional and ephemeral character of the experience of immersive theatre through alternative digital forms of presence. In doing so, these methodologies offer a means to better understand immersive theatre and its theatrical praxis, along with the properties of immersive experience and its potential as a way of knowing.

To this end, Josette Féral’s contribution to Archaeologies of Presence may be useful. Féral defines and sets out to understand the techniques for the manifestation of the ‘impression of presence’ or ‘presence effect’ (Féral 2012, 30-31). Féral describes this as ‘the feeling the audience has that the bodies or objects they perceive are really there within the same space timeframe that the spectators find themselves in, when the spectators patently know that they are not there’ (Féral 2012, 30–31). Through techniques that suspend the boundary between reality and imagination, the feeling of presence resides in sensation more than representation (Féral 2012, 34). Recreating the space time frame of Because the Night was therefore an important consideration in order to establish this presence effect.

In performance theorist Diana Taylor’s revisitation of her concepts of repertoire (as embodied knowledge) and archive (associated with history and preservation) within the digital age, she reflects on the new issues that digital technologies raise in terms of memory and knowledge production and how they constitute another system of transmission (Taylor 2012, unpaginated). She argues, with cyberspace ‘what counts as embodied knowledge has morphed’ and raises ‘new issues around presence, temporality, space, embodiment, sociability and memory’ (Taylor 2012, unpaginated). In exceeding print or the ‘flatness’ of video, 3D visualisation and extended reality technologies offer means to ‘both document and consult ‘live’ practices’ of performance in ways that capture kinetic, temporal and aural dimensions of the event/work (Taylor 2012). Archived mediations of performance may lead to new hybrid works of ‘live’ and online performances, which allow ways to access the experience of these works that would otherwise be unavailable.

While Rose Biggin’s first study of the immersive theatre company Punchdrunk’s work provides theoretical insight into the theatrical praxis of this contemporary performance form (Biggin 2017), there is still much to be learned about the theatrical logistics of how these complex performances are devised, designed, orchestrated and performed through embodied knowledge and practical experience. Punchdrunk’s documentary film Punchdrunk: Behind the Mask (Henderson 2023) provides some useful insight into the orchestration of the event with interviews of various members of the production team and revelation of the tools, such as a ‘loop chart’ and ‘master document’, which they use to track the different trajectories of performers and the multiple loops of a performance within one run.

Given the audience’s physical, imaginative, interactive and volitional involvement in such an immersive work, this form of immersive performance presents a unique challenge for archivists and its study. While the surrounding environment, with its hidden details and affordances, is critical to the storyworld and audience’s discovery of it, print and two-dimensional video documentation alone do not capture either the environment or way that the story is experienced. Advancements in three-dimensional and high-sensory technology allow the creation of digital archives that are performative and suited to the non-linear, multiple bifurcating storylines and enable the exploration of the multiple audience perspectives offered in immersive performance works. In producing an immersive archive, Malthouse and Deakin Motion Lab researchers explored how an immersive theatre production could be both documented for purposes of research, but also extended to provide alternative ways of accessing and experiencing the work.

Multimedia methods and materials to extending and reconstructing performance and performance spaces

There are a number of projects that have explored the deployment of 3D visualisation, augmented and virtual reality, computer animations, real-time walkthroughs and multimedia assets to reconstruct performance and performance spaces, such as the pioneering legacy project THEATRON (King's Visualisation Lab 2011), or the more recent virtual reconstruction of the Globe Theatre by a team of researchers from the University of Pisa, based on analysis of Elizabethan carpentry techniques and using AutoCAD software to create a version of the theatre that is historically reliable but also buildable (Croce et al. 2022). In creating an interactive archive of Because the Night (Greuter et. al. 2022) the aim was not to reconstruct the venue as such, but to create an archive of the performance that provided an embodied, immersive experience of it that could serve two purposes. Firstly, similar to the intentions of Croce et al.’s Globe reconstruction, the archive provides a resource for understanding how the immersive performance is built. It also serves as an extension or alternative venue of or way to experience the performance itself. The use of emerging augmented and virtual technologies to document or create an alternative digital version of a performance challenges ways of thinking about embodied knowledge systems.

Two existing approaches which use extended reality technologies to document or extend performance venues and events provide useful critical reference points for the Because the Night archive and for the different challenges that emerge with the development of immersive methodologies for documenting or extending immersive performance. The first is ‘The Visualising Lost Theatre Project’ (Tompkins et al. 2022), which uses VR technology to create visualisations of lost venues for embodied exploration of these spaces. The second is a remote and synchronous online companion version of Punchdrunk’s New York City production of Sleep No More created with MIT Lab (Torpey et al. 2012; Higgin 2012; Horn 2012). These projects highlight the unique challenges of developing immersive methodologies for documenting or extending immersive performance.

Immersive research method

As a method of visualisation ‘as’ research, Tompkins et al. argue the virtual reconstruction of performance venues using 3D visualisation technologies conveys an ‘immersive feel’ of the spaces, which in ‘The Visualising Lost Theatre Project’ enables active investigation. They elicit the space’s dynamic dimensions.

Conceiving of performance venues as ‘living systems’, the project points to the essential role that venues play in performance and how extended reality technologies may be deployed to provide new methods for re-presenting and researching the affordances, affects and ephemeral dimensions of performance that are generated by the environments in which they take place. As Tompkins et al. note the venue dictates ‘the scale of a performance, the special effects that are possible, the configuration of performer-audience relationships, how audiences move through them, and who mixes with whom in the process’ (Tompkins et al. 2022, 3). The virtual models allowed performers to ‘inhabit’ and work ‘inside’ the virtual reconstructions as a ‘close-to lifelike experience of a venue’ (Taylor 2012, 5). This enabled a means to gather embodied data of performers’ responses to the spaces, to test theories about the material conditions of the venues through embodied practice.

With the expansive understanding of what constitutes a ‘lost’ venue encompassing the disposable architectures of contemporary theatre production and itinerant cultural practices, the ‘Visualising Lost Theatres project’ provides a useful approach for the study of immersive theatre productions. In the case of Because the Night, the impermanence of the venue was felt even more acutely by the unexpected curtailment of the run due to pandemic lockdowns, which motivated the experimentation with alternative methods of re-presenting the production. In addition, as a dispersed non-linear performance form, immersive theatre has an intrinsic ephemerality. Each audience member chooses disparate spatial and temporal routes through the performance setting, which result in unique configurations of narrative. Therefore, with immersive theatre, the configuration of narrative could be added to those relationships of the living system that fade and are lost.

Tompkins et al. argue that models are ‘much more than stand-ins for the real thing’ (Thompkins et al. 2022, 7). They are accumulations of architectural knowledge, sites of cultural production and spaces for performance research, whereby the virtual space becomes a space for action and experimentation. This is the case with the interactive archive of Because the Night (Greuter et. al. 2022). It goes beyond being a surrogate experience, but also a visualisation. The audience is placed within the midst of the performance and can experience it visually, spatially and temporally.

Companion worlds

In contrast to Because the Night’s temporary production, Punchdrunk’s New York City production of Sleep No More has a longer term home in the repurposed warehouse fictionally named McKittrick Hotel, with a set of 100 cinematically detailed rooms. In 2012, the MIT Media Lab collaborated with the company to create a synchronous online companion world to the live experience in the New York venue. Online participants were partnered with live audience members to explore the performance together, but with personalised multimedia content that allowed the online participant to have their own experience of the production. However, this was co-created with their remote partner on the ground (Torpey et al. 2012). The project evolved in response to the problem that video and imagery is insufficient to capture the immersive experience of their work. As a solution, MIT Media Lab created a text-based virtual environment accompanied by a combination of immersive soundscape along with moments of live-streamed and pre-recorded video together with still imagery that evoked the spaces and the interactivity of the live performance.

The audience in the venue of the live performance was given a specially enhanced version of the masks that all audience members were provided with for the performance. The enhancements included physiological and environmental sensors and bone conduction transducers to produce sensor data shared between the online and on-site audience members. Their actions could trigger ‘actuated props’ installed in the space when they both inhabited the same physical and virtual space of the object. A show control system, Cauldron, could generate and stream content in both the physical and online world based on the interaction of the two audience members to create a unique interactive and responsive experience for both.

This approach provides a useful example of how a range of technologies and media can be employed to extend remote access to an alternative experience of a performance as it is happening, while evoking the interactivity and agency involved in immersive performance. As Higgin described it, the aim of the project was to create a cohesive and meaningful online experience ‘which lived up to the visceral intensity of the live show’ (2012, unpaginated). In doing so, the company sought to create a project with the reach and access of NT Live, but through ‘remote and real world interconnected theatrical immersion’ (Torpey et al. 2012). The result was not a model or recreation of the experience for the online audience, but an enhanced version for the on-site audience and a kind of text-based adventure game combined with imagery, live-streamed and pre-recorded video. With Because the Night we set out to provide that visceral intensity within a virtual visualisation that allowed the experiential exploration and navigation of the performance space, actor’s performances and the narrative objects.

Approach and results: Because the Night interactive archive

The team behind the interactive archive of Because the Night (Greuter et. al. 2022) aimed to use a game engine and 360-degree video to create a digital version of the live immersive theatre performance. The research objectives were to holistically capture the experience of the immersive theatre performance for a digital audience, replicate key affordances of the original performance, explore the potential of emerging technologies and consider game engines as a sustainable platform for presenting and documenting immersive theatre performances.

To create an accurate archive of the performance, it was important to capture not only the spatial dimensions of the venue, but also the atmosphere and emotional intensity of the experience. This involved including the kinetic and auditory aspects of the work and allowing the audience to interact with performer-audience relationships, explore narrative objects and experience the temporal space of the performance. The digital archive needed to convey embodied knowledge.

The project captured 360-degree videos of the performance, music and prop images, as well as 360-degree photographs of the set, all integrated into a game engine. This process resulted in 102 sound sources, 332 photographed artefacts, 257 audience viewpoints and 9 hours of film footage, which were processed and incorporated into the game engine. The team chose Unreal Engine for its powerful visual Blueprint scripting tool and royalty-based licensing model, which minimised ongoing development and maintenance costs.

The immersive archive enables users to manipulate the 360-degree perspective, follow characters, scrutinise rooms, locate and investigate narrative objects and decipher secret codes to access two locked rooms featured in the live performance. A floor plan of Elsinore displaying the actors’ positions within the performance's sixteen rooms was also included in the archive. The system effectively replicated several of the original performance's affordances, including environmental exploration, spatially emergent narratives, temporal constraints, object examination and discovery and interactive environmental challenges.

Performance modes

Because the Night differed from traditional theatre performances in that it featured six continuous and simultaneous 90-minute performances, each following a different character, which intersected with other characters’ performances at different points throughout the show. To enhance the digital experience and accommodate various user preferences, three gameplay modes were implemented, each with its unique timing rules.

The Authentic mode allows time to progress continuously as a 90-minute performance, with all six performances running simultaneously, even when the audience pauses to examine props. Rooms on the map are hidden until the player visits them, allowing players to construct a spatial understanding of the set over time. Secret rooms are hidden from the player until the player finds a code to enter them. This mode effectively conveys the ephemerality and spacetime frame of the live performance, the pressure and influence of time on audiences' decision-making within the performance of how they spend their time, and where they place their attention within the constraints of the performance duration.

The Standard mode offers a balance between an immersive experience and allowing the player more time to explore the clues in a room without missing key moments in the narrative. Time is paused when users study props or consult the map. Similar to the Authentic mode, all rooms are hidden from the player until they have explored them.

The Full Control mode grants the player complete control over the passage of time, enabling them to view all scenes, jump back and forth through the narrative, and take their time exploring the intricate details of the performance. All rooms are instantly accessible to the player. This mode provides the most flexibility for those who want to thoroughly examine every aspect of the production.

The Standard and Full Control modes are ideal for members who have seen the original play or players who completed the Authentic mode as they offer the opportunity to step out of the time flow and investigate aspects of the performance more closely, delving into the storyworld's detail, narrative trajectories and construction. The Authentic mode manifests the important temporal and spatial properties that contribute to an immersive experience. The temporal constraint or time pressure can heighten the engagement and sense of agency, knowing that decisions and strategies must be made and they have consequences. While this constraint collapses time or perhaps suspends it, it simultaneously expands the spatial dimension of the storyworld. There is more to be seen and explored than the time allows, creating the presence effect of a far-reaching world.

Narrative

To capture Because the Night for the interactive archive (Greuter et. al. 2022), the team needed to film the entire production. To facilitate filming, the team used an extended story flowchart and the original script to break down the performance into individual scenes. Each scene or shot was assigned a unique identification code and grouped based on their function, which included multi-character scenes, single-character voice-over scenes, idle scenes for timing purposes, ghost scenes for single audience-performer interactions and transition shots to capture performers’ movements between locations. An excerpt of the story flow chart is shown in Figure 2. Since the performances were simultaneous and continuous, it was important to carefully plan and execute the filming to capture each character's story and interactions with other characters.

Figure 2. Excerpt of story flow chart denoting character paths and timings

Filming 360-degree video

To accurately capture the immersive experience of the theatrical production, it was necessary to use a 360-degree camera with exceptional low-light performance. The Insta360 Titan Pro was chosen for its ability to record at the highest resolution possible for a single device while maintaining superior performance in low-light conditions, which are characteristic of theatre environments such as Elsinore. However, technical testing revealed that augmenting the theatre lighting during filming was necessary to achieve optimal video quality while preserving the original visual aesthetics of the play. Consequently, lighting presets were adjusted to increase brightness by approximately thirty percent, resulting in a clear, grain-free image. This camera, combined with the appropriate adjustments to lighting and post-production colour grading, allowed for faithful replication of the original performance's lighting and atmosphere in the digital archive.

To capture the multiple performance trajectories within the overarching performance, the filming routine began by positioning the 360-degree camera in the most suitable position for the scene. Rehearsals were conducted to make any technical adjustments, such as lighting or audio, as well as changes to performance or blocking before recording commenced (see Figure 3). Breaking down the continuous performance into individual scenes facilitated the capture of each performance in every room on the stage. The 360-degree video format facilitated the holistic capture of the performance in each scene. To enable filming, the film crew set up the recording equipment in the adjoining rooms on the set (see Figure 4).

Figure 3. The performers rehearse on stage with the 360-degree camera between them. Capturing an intimate angle of both performers talking ‘through’ the audience.

Figure 4. Crew setup in the adjoining room.

360-degree video post-production

During the post-production phase, the development team utilised Mistika as the primary software suite to process and stitch the 360-degree video footage. The raw footage, consisting of multiple videos from the 360-degree camera, was imported into Mistika for previewing and editing the stitched footage. This allowed for the evaluation of stitching quality and any required modifications to be made (see Figure 5). Finally, the stitched 360-degree video was exported as a video and imported into Adobe Aftereffects for additional adjustments such as colour grading and audio synchronisation, which ensured a consistent and immersive experience.

Figure 5. 360-degree video post-production.

Figure 6. Group assembly of the performance.

Figure 7. Position markings of 360 images.

Narrative flow replication

To enable an emergent narrative for a digital audience using a game engine, the captured narrative sequences needed to be edited so that the narrative streams coincided at the correct time. During the live performance of Because the Night, this was achieved through ‘hidden triggers’ installed throughout the set, which could be activated by the actors to cue other performers in different spaces, allowing the performance to adapt to specific performance conditions and audience interactions, resulting in finish times varying by up to 30 min depending on the night. During the recording phase, ‘idle’ scenes were recorded that could be looped, stretched, or shortened during the editing process. The intersection of the narrative paths of two characters at the correct time was achieved by adjusting the length of the preceding idle scene. Combining all the scenes into a cohesive timeline was essential for exporting the scenes as location-specific timelines to enable user agency in the game engine. This iterative editing process preserved the multi-narrative story, which was then transferred into the game engine, where the audience’s movement could be reconstructed. Figure 6 illustrates the group assembly of the performance.

Environmental exploration

To recreate the audience's ability to move around within a simulated environment with the performers, the team employed a game engine to manage temporal elements, while environmental exploration was captured through a series of 360-degree images, akin to Google Street View.

The process involved meticulous geospatial positioning and recording to ensure an accurate spatial reconstruction. The camera was placed at strategic positions to follow typical audience members’ movements, with each location marked on a CAD drawing of the set. The camera's orientation was aligned with magnetic north, allowing for a relative point-to-point angular relationship between positions critical to reconstructing the environment in three-dimensional space. The method for movement involved blending two inverted spheres that would alternate positions as the audience member moved between them throughout the stage.

The subsequent process of capturing the 257 still images required careful positioning and recording. The camera was positioned strategically, with each shot taken from a marked location on the CAD drawing. This process ensured accurate spatial reconstruction, allowing for seamless exploration of the environment in the game engine. Figure 7 shows the placement of real-world cameras on the set plan, which were later positioned on the 3D model in the game engine.

To create a digital replica of the original performance that captures the immersive experience, the team employed 360-degree photography to replicate each space as a series of navigable still views. Compared to traditional two-dimensional photography, this technique better captured environmental exploration and eliminated the impact of framing on perception. Furthermore, alternate images for temporally changing environments helped preserve the sense of forward motion and the audience's sense of being a silent witness in an unchangeable narrative. Ultimately, this captured imagery became a fully navigable and interactive environment complete with scenes, props and freedom of movement.

Environmental changes were also replicated in certain positions to correspond with alterations in the narrative, enabling audiences to explore the environment and consider subsequent scenes and events regardless of whether they were observed during the original live performance or in the digital version. Figure 8 depicts the same scene with environmental changes.

Figure 8. Environmental changes: before and after a scene.

Interaction

In order to recreate the audience's interaction with the various dramaturgical objects in the digital version, a workflow and interaction-design framework was developed that utilised studio photographs of each object. These 332 objects, which ranged from a diary to an illustration to a weapon, provided additional information and narrative affordances through their discovery. The interaction system allowed players to access the objects through a look-and-click method, and they could also click and drag the objects around their screen and zoom in with the scroll wheel. This approach aimed to preserve the tactile agency of the audience when inspecting the dramaturgical objects, while also improving information transfer by enabling greater control over viewing position.

In order to maintain the original aim of the labyrinthine set design of Because the Night and to ensure that it was not lost in the digital version, a map was included. The completed map design (see Figure 9), complete with characters, allowed the digital audience to navigate Elsinore through the map and to access scenes by clicking on rooms with characters (see Figure 10).

Figure 9. Completed Map design with figurines on game board (designed by Dennis Ioannou).

Figure 10. Finished map and figurines on game board (designed by Dennis Ioannou).

Research outcomes

The research aimed to develop an immersive archive of the Because the Night theatre performance that closely simulated the experience of the original audience. This was achieved through a game engine that incorporated 360-degree video and still images, as well as interactive elements such as object examination and environmental exploration.

Environmental exploration was captured through a series of 360-degree images, with meticulous geospatial positioning and recording to ensure accurate spatial reconstruction. Object examination and discovery were also included in the digital experience through a look and click interaction system using studio photographs of each object or page.

The set's labyrinthian design was preserved through a map that enabled users to navigate Elsinore and access scenes by clicking on rooms with characters. Three gameplay modes were developed to allow users to experience the performance in different ways, with one mode replicating the original performance's temporality and its effects on agency and ephemerality.

Overall, the immersive archive successfully preserved the essential characteristics of the original performance, such as the ability to move around the environment with the performers and the immersive experience of exploring an expansive world. The inclusion of interactive elements and different gameplay modes also allowed for a more nuanced and flexible experience for digital audiences.

Conclusion

The Because the Night Immersive Archive is a significant accomplishment in digital archiving for immersive theatre. By utilising a game engine, 360-degree video and photography, the project team was able to capture and replicate the essential spatial and narrative affordances of the original performance in a digital format. This success was achieved through meticulous planning, capturing, editing and post-production work, which allowed for the best simulation and understanding of essential immersive theatre characteristics.

The use of game engine technology enabled the digital audience to interact with the environment, objects and characters, and follow multiple narrative trajectories, resulting in an immersive digital performance experience that can be accessed from anywhere. Additionally, multiple game modes allow the digital audience to customise their engagement and flexibility when experiencing or re-experiencing the performance. This is particularly significant in the context of immersive theatre, where individual audience engagement and agency are critical to the experience.

Moreover, this project is not just a significant accomplishment in documenting immersive theatre but also serves as a starting point for future research and development. As with other performance archiving methods, technology's rapid development and change have led to new affordances and challenges in scaling, replicating, or applying specific methods to other performances. Additionally, the longevity and availability of required software pose challenges to the sustainability of digital archiving projects. Finally, this work required the significant effort of large teams with diverse skills and expertise, which is another factor to consider in future innovation in this field.

In conclusion, this project is a valuable contribution to ongoing efforts to create new embodied research methods that allow for a wider audience to understand and experience the unique characteristics of immersive theatre. It also highlights the importance of immersive experience as a powerful means of knowledge acquisition.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Stefan Greuter

Stefan Greuter is Professor of Screen and Design at Deakin University and the Director of the Deakin Motion Lab, lead CI and designer of the Interactive Archive for the Malthouse Theatre. In his research Prof. Greuter addresses problems through design thinking, interaction design, game design and the creative use of emerging technology to develop solutions that result in serious games applications and playable user experiences that promote immersive playful interactions.

Gerard Mulvany

Gerard Mulvany is a PhD candidate at Deakin University and a researcher with the Deakin Motion Lab. Gerard's dissertation in partnership with CSIRO and Malthouse theatre, explores how emerging technologies can facilitate future metaverse hybridisation of theatre.

Misha Myers

Misha Myers is Professor of Contemporary Performance and Head of the School of Stage and Screen at University of Greenwich, an Honorary Professor at Deakin University and CI on the Documentary Team for the Interactive Archive. As a digital performance researcher and maker, she has cocreated methods with diverse partners in global contexts for sharing people's stories, practices and embodied knowledge of placemaking through immersive, cross-platform and multimedia artworks and archives.