Applications of complex narratives

ABSTRACT

This paper aims at providing an overview on IDNs that address complex issues to gain a better understanding what design decisions have been taken in those prototypes and applications and then to make the resulting knowledge available to practitioners for the representation of complex topics. In our work, complexity refers to a significant societal challenge or current complex phenomenon, such as global warming, globalisation, crime, colonialisation, and so forth. Complexity in this context has been defined as a problem that characterises the behaviour of a system whose components interact in multiple ways and that allows various ways of interpretation and validation of such interactions. The paper corresponds closely to the work undertaken by the COST action INDCOR, especially WG1: Design and development.

KEYWORDS:

• Word
• IDN design
• IDN applications
• IDN developments

1. Introduction

Storytelling plays a central role in human culture through the use of narrative as a cognitive tool for situated understanding (Gerrig, 1993). This ability to organise experience into narrative form and structure has been applied across a range of practises, such as entertainment, education or training (Chaitin, 2003; Sturgess, 1992), and various forms of media (Bordwell, 1985; Gloag & Beard, 2009; Lodge, 1992; McCloud, 2006; Schell, 2008).

One of the most compelling applications of narrative intelligence is interactive digital narrative (IDN). IDN applications and prototypes have the potential for representing, experiencing and comprehending complex phenomena (Koenitz et al., 2015; Murray, 1997). IDNs are digital experiences in which users create or influence a dynamic storyline through actions, either as the protagonist of the unfolding storyline (as in digital drama) or as an observer who can navigate the story space aided by a system. The goal of interactive narrative is thus to immerse the user in an intellectual as well as emotional experience so that the user’s actions can have a direct impact on the direction or outcome of the storyline. Thus, the user is given the opportunity to “replay” the story in different forms. Replay will provide reflections of the various, often opposing, directions and meanings the story might have, depending on the starting point and interactions (Mitchell & Kway, 2020).

The application of interactive digital narratives to the presentation of complex issues, such as COVID-19, racism, globalisation, global warming, the current war in Ukraine, etc., is seen as a monumental endeavour, because it addresses complexity on several levels: content selection, mode of interaction, audience perception and narrative generation. For example, the understanding of natural hazards and their impact on human lives requires knowledge on their physical and statistical behaviour that is best understood through narratives that facilitate the reflection on solutions for prevention and emergency actions in a known context and environment (please see, e.g. Havlik et al., 2015). In such instances, narrative representations of complexity are required that can make use of digital tools “to create representations that contain competing perspectives, offer choices, and show the resulting consequences while allowing for repeat experiences” (Koenitz & Eladhari, 2019).

Seen as such, IDNs form a triad composed of creators, active participants and dynamic narrative mechanisms that are expected to:

• Contain many different—even competing—perspectives in the same instance. This will enable observers to look at phenomena from various point-of-views.

• Promote active participation of both end-users and stakeholders and facilitating self-directed experiences.

• Encode complex dependencies and allowing them to be experienced as narratives.

The COST action INDCOR (Interactive Narrative Design for COmplexity Representations)¹ is currently working on a comprehensive view of Interactive Digital Narratives (IDN) as a field of study and a design discipline, including work on a shared vocabulary, authoring tools, validation and assessment of IDN prototypes. For IDNs, INDCOR has a particular focus on the representation of complex issues. This requires an interdisciplinary approach of understanding IDNs to address complexity as a societal challenge by representing, experiencing and comprehending complex phenomena.

This paper aims at providing an overview on IDNs that address complex issues to gain a better understanding what design decisions have been taken in those prototypes and applications and then to make the resulting knowledge available to practitioners for the representation of complex topics.

2. Inventory design

The aim of the presented work is to establish an inventory of recent and current IDNs, to be able to study the design of these to gain insight in current design practices. The primary source of the IDNs was within INDCOR taking advantage of the internal knowledge within the network. In addition, the inventory is grounded on search with various engines (i.e. Google,² Bing,³ CC-Search,⁴ ACM DL,⁵ docubase⁶), using terms such as: IDN complex applications, narrative-focused games, interactive documentaries, journalistic interactives, complexity installation pieces, XR experiences and narrative interfaces to big data. The terms have been selected based on the keywords used in the descriptions of the applications that have been suggested by the INDCOR community. We filtered out findings that addressed systems for the analysis of complex narrative structures (e.g. Murray, 2017–18; Strezoski et al., 2018, environments that have a fictional focus (see Ivan Sanchez-Lopez, et al., 2020 as a good collection of those). We excluded systems that have been developed before 2000 (even though we admit that interesting work has been done also before, as exemplified by the MIT Interactive cinema group⁷). Works have been selected that allow its audiences to experience the consequences of a series of choices and reconsider these choices through replay. Through applying these criteria, the number of works was limited to 18 applications described in Section 3. We did at the time also consider authoring tools or design specifically as criteria, however, chose to omit this in the first version and rather use this as grounds for future work and additional inventories to be developed.

3. Inventory

The below inventory contains 18 applications, which are presented in the following way. The works are increasingly ordered based on the production date, so that technical developments can be addressed. We also tried to group the applications based on the presentation media and the domain the work addresses. Each work also includes a short content description and a link to the main web resource. The list should not be considered as conclusive and will be updated during the remaining time of the INDCOR project.

3.1. Vox Populi (2006)

Domain: Interview representations on controversial topics

Form: Interactive documentary

Medium: Video in a desktop environment

URL: https://homepages.cwi.nl/~media/demo/VoxPopuli/

Vox Populi is a system that automatically generates video documentaries. The application domain is video interviews about controversial topics. Via a web interface, the user selects one of the possible topics and a point of view she would like the generated sequence to present, and the engine selects and assembles video material from the repository to satisfy the user request. The video material is mainly interview material. The system uses rhetorical annotations to specify a generation process that can assemble meaningful video sequences with a communicative goal and an argumentative progression. The annotation schema encodes the verbal information contained in the audio channel, identifying the claims the interviewees make, and the argumentation structures they use to make those claims.

3.2. HIghrise (2009–2013)

Domain: Urbanism

Form: Interactive documentary

Medium: Mixed media, linear video

URL: http://highrise.nfb.ca/

An Emmy-winning, multi-year, many-media, collaborative documentary experiment at the National Film Board of Canada, that explores vertical living around the world. HIGHRISE has generated many projects, including mixed media, interactive documentaries, mobile productions, live presentations, installations and films. HIGHRISE explores how the documentary process can drive and participate in social innovation rather than just to document it.

3.3. Aspergion (2010–2011)

Domain: Training for respecting teens with Asperger’s Syndrome

Form: Educational role-playing game

Medium: Animated cartoon, text communication

URL: https://vimeo.com/showcase/4673180

Aspergion is a digital game to promote respect for people with Asperger’s Syndrome. The game is intended as a learning resource for neurotypical (normal) secondary school students. Asperger’s Syndrome is an autism spectrum disorder at the least severe, or high-functioning, end of the autism spectrum. The plot of the game is to save the Aspergion galaxy from being pulled into a black hole, by focusing on the positive emotions. According to the game, the planets of Aspergion were out of harmony, resulting in conflict due to their inhabitants being intolerant and exclusionary, and communicating poorly. The goal of the game for an individual player was to solve these conflicts by growing one’s avatar from being a rogue space traveller to an “Aspinaut Admiral, Grand Master of the VIBE”.

3.4. Bear 71 (2012)

Domain: Species observation

Form: Interactive documentary

Medium: Video, audio, interactive graphics

URL: http://bear71.nfb.ca/#/bear71

Bear 71 is an interactive National Film Board of Canada web documentary by Leanne Allison and Jeremy Mendes about a grizzly bear in Banff National Park, who was collared at the age of three and was watched from 2001 to 2009 via trail cameras in the park. Bear 71 examines the story of the bear through the digital interactive medium, creating a technological interpretation of nature for the user to explore, and for the bear to inhabit as she tells her story through 11 starting points.

3.5. Fort McMoney (2013)

Domain: Economic development/Urbanism

Form: Interactive documentary/strategy video game

Medium: Video, audio, interactive graphics

URL: https://docubase.mit.edu/project/fort-mcmoney/

https://www.youtube.com/watch?v=uUfaXPb4puo

This work is about Fort McMurray, Alberta, Canada and Athabasca oil sands development, directed by David Dufresne. The documentary uses the structure, tropes and timing of a video game, to facilitate users to decide the city’s future and attempt to responsibly develop the world’s largest oil sands reserves. The game is played out in phases, which makes the project live at certain times and inactive between phases, driving audiences to moments of shared interactivity and play-through, with each phase featuring new content. While an influx of outsiders continually mixes with the local contingent, Fort McMurray is also the ground for debates between big business representatives who want to push the oil industry forward, environmentalists who see the ecological dangers of exploiting Canada’s oil sands, and the politicians who are embroiled in key decision-making processes.

3.6. Yasmine Adventures (2013–2015)

Domain: Cultural Heritage

Form: Spatial Narrative Installation

Medium: Location-based mixed media

URL: https://www.valentinanisi.com/Yasmine-Adventures

The Yasmine Adventures project is concerned with the design and evaluation of a mixed reality application aiming at extending the experience of a museum into the surrounding of the Meringplatz neighbourhood in Berlin. Implemented as a mobile application Yasmine Adventures engages visitors to explore neglected urban areas across a three-layer mixed reality interface that includes the opinions of community members and other contextual information via fictional storytelling.

3.7. Lucid Peninsula, DreamScope (2013–2015)

Domain: Cultural Heritage

Form: Physical Narrative Installation

Medium: 360° virtual reality components

URL: https://futurefabulators.m-iti.org/projects/dreamscope/

DreamScope gives the audience the opportunity to have a glimpse into the Lucid Peninsula. The audience can be involved in the Time’s Up physical narrative of the lucid peninsula through a telescope that enables them to peek into a virtual reality 3D world—DreamScope Viewer. Such custom-made 3D world portraits how the Lucid Peninsula world looks like, with its dusty atmosphere, GIN distilleries and nests. On the other side, visitors can access some of the dreams of the Lucid inhabitants through the DreamScope Catcher, which provides a location aware mobile application where the Lucid Dreams will be available in three different locations within walking distance from the exhibition place.

3.8. A dictionary of the revolution (2013–2015)

Domain: Political conversation

Form: A polyvocal interactive documentary

Medium: Interactive graphics, text

URL: http://qamosalthawra.com/en

This is an experiment in multi-vocal storytelling. It is a series of 125 texts woven from hundreds of people who were asked to define the evolving language of the Egyptian revolution. The project was designed by Amira Hanafi to document the rapid amplification of political conversation in public following the uprising of 25 January 2011. To engage a large and diverse group of people in the documentation, she created a vocabulary box containing 160 words in the Egyptian colloquial language that were frequently used from 2011 to 2013 in public political conversations. In 2014, the box was used in around 200 conversations with individuals across six governorates of Egypt. Choosing cards from the box, people talked about what the words meant to them, who they heard using them, and how their meanings had changed since the uprising. Using transcription formed the basis of the lexicon.

3.9. The Quipu project (2015)

Domain: History

Form: Interactive documentary

Medium: Interactive graphics, audio

URL: https://interactive.quipu-project.com/#/en/quipu/intro

Hundreds of thousands of Peruvians were sterilised in the 1990’s in Peru. Many claimed this happened without their consent but have been repeatedly silenced. The project’s name, design and inspiration come from the ancient Inca system, Quipu, a system of colourful knotted strings used to keep official records and tell stories. Users click through coloured dot icons, each representing a section of the testimony, and listen to the audio of phone calls from over 100 women who dialled a collected telephone number and recorded messages about their experiences. The audio, along with responses from listeners, was collected using Drupal VoIP, an open-source technology developed at MIT Centre for Civic Media.

3.10. Pregoneros or Medellin (2015)

Domain: Street culture

Form: Interactive documentary

Medium: Interactive 360° video

URL: https://pregonerosdemedellin.com/#en

This project was created to highlight Colombian popular culture and street vendors. In the interactive documentary, users move around the city by scrolling through street walk imagery, and gradually encounter five different street vendors. The soundscape of the city plays an integral part in the experience, as users are alerted that vendors are nearby when they hear their voices.

3.11. Miner’s Walk (2015–2016)

Domain:

Form: Interactive documentary

Medium: Interactive 360° video

URL: http://www.minerswalk.com/

Miner’s Walk is an interactive documentary exploring the sulphur miners of the Ijen Crater, a group of Indonesian workers who collect heavy sulphur rock, trekking up and down the steep slopes of the volcanic crater. In recent times, the Ijen Crater has been transformed into a destination for domestic and international tourists. An increased tourist presence and contentious mining practices by the sulphur mining industry have led Ijen into becoming one of the few places in the world where miners, the sulphur industry, and tourism exist side-by-side in an uneasy fellowship. Miner’s Walk is an experiment in interactive storytelling and merging of short and longform video with an online interactive experience. It grants viewers access to the miner’s world, following the miners on their precarious journey into the Ijen Crater as they collect and carry sulphur rock and let the miners themselves describe the reality of their work in relation to the impact of the technology, and how tourism might shape their future.

3.12. Hearts and minds: the interrogations (2016)

Domain: Human Rights Abuse

Form: Interactive documentary

Medium: Virtual reality, 2D version for cinema version

URL: http://www.crchange.net/hearts-and-minds/

A VR CAVE work/Unity application designed as an interactive, immersive and cinematic environment that draws users into the memories of ordinary American soldiers who became torturers while serving their country. This project is based on the testimonies by human rights foregrounds veterans of the US military about enhanced interrogation practices and human rights abuses during the Iraq War.

3.13. TOXI-CITY: a climate change narrative (2016)

Domain: Climate Change

Form: Interactive documentary

Medium: Panoramic cinema (cinemascope), virtual reality environments (CAVES), video walls and multi-monitor installations

URL: http://www.crchange.net/toxicity/

Set in the US Eastern seaboard in 2020, this film follows six fictional characters whose lives have been transformed by the rising of sea level and flooding in an urban and industrialised region of America’s North Atlantic Coast. Fictional testimonies are set against nonfictional accounts of actual deaths that occurred during Hurricane Sandy and other recent storms and floods. This is a combinatory narrative film that uses algorithms to draw fragments from a database in changing configurations every time it is shown. As some stories seem to resolve, others unravel. Just as with the conditions of ocean tides and tidal shores, the stories cycle and change without clear beginning or end. These offer moments of resolution, contact and visions of the future, before the narratives are broken apart and a fresh cycle begins.

3.14. Gaming for peace (2016–2019)

Domain: Training in conflict prevention and peace building

Form: Educational role-playing game

Medium: Animated cartoon, text communication

URL: https://gap-project.eu

https://www.youtube.com/watch?v=PkfGbVr5ekc

Conflict Prevention and Peace Building (CPPB) missions come from diverse organisations and nations and must coordinate in the temporary network or umbrella organisation that comprises each CPPB mission. Even if the structures needed for coordination are in place, diversity in organisations (militaries, police forces, civil organisations), as well as gender and culture (national, ethnicity, religion) aggravate the understanding of diverse personnel and effective communication and cooperation in contexts of diversity. Gaming for Peace (GAP) is developing a curriculum of soft skills derived from interviews with experienced military, police and civilian peacekeeping personnel. GAP is embedding a selection of soft skills related to peacekeeping in a digital role-playing game with in-game assessment. The key soft skills are gender awareness, cultural competency, communication, cooperation, decision-making and stress management. In-game assessment is reported in individual “skills passports” for players, and the learning metrics have been standardised against international benchmarks.

3.15. Memories of Carvalhal (2017–2020)

Domain: Cultural Heritage

Form: Location-based game/interactive digital narrative

Medium: Augmented reality

URL: https://mocp.m-iti.org/

This is a multidisciplinary project of the Natural History Museum of Funchal. The purpose of the game is to study the engagement of teenage audiences in museum contexts, as this generation is identified as an audience group that is often excluded from a museum’s curatorial strategies and appears to be generally disinterested in what museums might offer. The game comprises two experiences targeted at teenagers. These experiences are set in the same museum and based on the same content, although presented in distinct formats. One experience is a bespoke story and game interactive experience (Haunted Encounters). The other experience is driven by the story of Turning Point in form of a fictional nineteenth century drama with real-life characters that provides two storylines with two distinct points of view of the tragic story of Marina and Xavier.

3.16. Fragments of Laura (2018)

Domain: Cultural Heritage

Form: Location-based game/interactive digital narrative

Medium: Augmented reality

URL: https://mocp.m-iti.org/

Fragments of Laura are a transmedia experience to raise tourist awareness about local values, such as biodiversity, habitats and local costumes. The story takes place in Funchal during the eighteenth century, when it served as a busy port for transatlantic trade. Laura Silva discovers and carefully documents the unique properties of Laurisilva forest. Using mobile devices (smartphones and tablets), the visitor can explore a historical part of Funchal. The mobile tour provides visitors with a way to explore some of the locations in Funchal and learn about different historical times in Madeira. The interviews in a web-based journalistic style provide in-depth current information about the island’s most precious values.

3.17. Kampen om Maden (2018–2021)

Domain: Cultural Heritage

Form: Location-based augmented-reality game

Medium: Location-based augmented reality

URL: https://play.google.com/store/apps/details?id=com.hauntedplanet.branded.mosedefort

https://api.hauntedplanet.com/casebook/show?uuid=0d54879c-b0bc-4289-a36b-900f19ff7f2f

A location-based augmented-reality game for smartphones that engages with political and cultural themes during WW1 in Denmark. Museet Mosede Fort Danmark 1914–1918 is a World War 1 museum based in Greve, Denmark. The game tells a complex story about how national policy decisions during the First World War, especially around the scarcity of food, led to the development of ideas and thinking that became the foundation of the Danish welfare society and are still relevant today. The game experience is structured around nine double encounters. Of each double encounter, the first brings to the fore a particular problem related to food shortage and malnutrition that arose in Denmark. The second part of the double encounter is a “companion encounter” that relates to the same theme but shows how new solutions to the food-related problems arose from the war and formed new ways of organising the Danish society.

3.18. Home of the trolls (2020–ongoing)

Domain: Tourism

Form: Interactive digital narrative

Medium: Augmented reality/smart devices

URL: https://drive.google.com/file/d/1gFHe6EO3WcI3wmZQRHKpkaRjtJzZnyAz/view?usp=sharing

A location-based storytelling for adventure tourism that utilises commercial smart devices to facilitate interaction with the environment to make the local folklore and mythical stories available on site.

3.19. World of wild waters (2020–ongoing)

Domain: Science

Form: Narrative-focused game

Medium: Virtual reality

URL: https://www.woww.no/

This is an interactive virtual reality prototype to explore natural hazards based on real data, realistic scenarios and simulations. The aim is bringing together knowledge on physical and statistical behaviour of Natural Hazards and knowledge on digital storytelling and human behaviour to create immersive user experiences for educational settings.

4. Discussion and challenges

This paper is intended as an overview and introduction to IDNs by examples in an inventory rather than an in-depth analysis of the design decisions and explanation of the chosen examples. However, there are a few issues and challenges highlighted by the presented inventory highlights that warrant future work before being able to provide knowledge for practitioners. Most importantly, our findings indicate the need for looking into new specific methodologies, tools and practices intended for designing and implementing IDNs that address complex issues. The following discussion aims to identify the areas that need further research and innovation.

Although the domains of the applications outlined in Section 3 are rather broad, it is apparent that two forms of applications dominate. The most popular format is the interactive documentary, followed by games in various forms. It is understandable that the documentary genre plays such a favourable role, as a documentary aims at instructing, educating and reflecting about reality issues in the past, now and even in the future. The points of view provided as well as the topics addressed by documentaries are multi-faceted if not complex (Nichols, 1991). Moreover, documentary is an established way for critical discourse and structurally facilitates, mainly in films, a clear process for the presentation of the ideas to be communicated and the way how the material should be gathered and how it can then be arranged (Rabinger, 1998). In that sense, the presented works are examples of a classical documentary style, but their communication depth goes beyond the adaptation for the digital medium, as can be seen, for example, in Cybertexts (Aarseth, 1997; Coover, 2003; Moulthrop, 2003; Nelson, 2003) or the Evolving Documentary projects by Glorianna Davenport’s MIT Interactive Cinema Group.⁸

It is also not a surprise that in the inventory provided in Section 3 games feature prominently because of their inherent interactive nature. Games constitute a well-established medium for the experiential exploration of complex environments, and their mechanics are designed to integrate storytelling as a form of control (Schell, 2008).

Yet, it is interesting to observe that although the different works function in different media spaces, ranging from 2D text graphics to full virtual reality environments, the actual communication flow resulting in hermeneutic processes in the audience (Roth et al., 2018) is based on the underlying argumentation engines. This is rather easy to observe in an application like Vox Populi (Section 3.1), where the Toulmin argumentation model forms the core of the system. In other applications, such as Fort McMoney (Section 3.5), a dictionary of the revolution (Section 3.8) or Kampen en Maden (Section 3.14), the underlying argumentation or reflection structure, which drives the material selection, might not be immediately observable. Yet, it is noticeable that in the design of the systems, care has been taken that the internal representation of the material would be aligned with the design of the presentation engine. This requires the development of new technologies, which is particularly exemplified by Pregoneros or Medellin (Section 3.10). In this application, the creators aimed to realise a virtual street walk because the story is about the stroll through the streets, involving the urban soundscape, the surprise of sounds and the meeting of the street vendors. Since the story was meant to be an all-encompassing experience, the narrative value could only be materialised by transforming this story into an all-embracing interactive experience built with new software developed by means of HTML, CSS, Javascript, and a bike + a GoPro⁹ camera and microphones (Durand, 2015).

It can also be observed that over the years nearly all the works presented in Section 3 have been produced by interdisciplinary teams. This implies that those types of applications, even though designed and implemented with the existing authoring tools,¹⁰ need different types of authoring environments. Such novel environments should facilitate fragmented, distributed and interactive authoring that still results in a coherent narrative experience that potentially spreads across media, and hypothetically even predicts the audience’s potential intellectual and emotional state at any time. This kind of authoring can be described in terms of Koenitz and Eladhari (2021) as “narrative system building”. Moreover, such innovative authoring tools should be accompanied by systematic methodological design approaches focusing on the functional (logos), quality (ethos) and emotional (pathos) aspects of IDNs (Estupiñán & Szilas, 2020; Green et al., 2021; Irshad & Perkis, 2020; Marshall, 2021; Taveter & Iqbal, 2021). In this context, it also needs to be further investigated how the argumentation, communication and story models developed in linguistics and communication studies can be integrated into the ideation, authoring and generation processes, so that appropriate forms of reflection can be established in real time. This type of research leans more towards the knowledge representation side of such types of creative systems (see, e.g. Nack, 1996; Szilas et al., 2012).

A weakness of the provided inventory is that it does not tell a lot about the validity of the presented works, as there is not much information available on the number of users who experienced the applications presented in Section 3. Neither is available detailed reflective discourse material in the form of reviews or experience reports. This indicates that the field of IDNs is short of appropriate validation methods, like the ones put forward by Roth and Koenitz (2016), which are important so that those type of applications would become less niche and artistic and would be used by larger populations.

5. Conclusions

IDN applications rely on the process of design, authoring and distribution to users. To be able to understand this process, we have established an inventory of IDNs through the INDCOR COST action. This inventory provides examples of current practices with respect to designing IDNs for addressing complex issues and hence serves as an inspiration and guidance for the research community. The presented IDNs also point towards the need for looking into new specific methodologies, tools and practices intended for designing and implementing IDNs that address complex issues. In particular, we identified two specific areas that need research and innovation: (i) the methodologies for the design of IDNs; and (ii) the authoring tools aligned with the methodologies that address interdisciplinary and collaborative work and facilitate the collection and maintenance of story material in various media in combination with the related IDN engines.

Acknowledgements

The authors thank the COST Action 18230—Interactive Narrative Design for Complexity Representation (INDCOR) and its members, for the many thought-provoking debates on the topic of interactive digital narratives, complexity and representation. We especially thank all the members of INDCOR Working Group 1: Design and Development for their contributions on the inventory of IDNs. The research work by the second author has been partially funded by the European Social Fund via the IT Academy programme.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by European Cooperation in Science and Technology [grant number MoU—047/19].

Notes

1 https://indcor.eu/.

2 https://search.google.com/search-console/about.

3 https://www.bing.com/.

4 https://oldsearch.creativecommons.org/.

5 https://dl.acm.org/.

6 https://docubase.mit.edu/.

7 https://ic.media.mit.edu/.

8 https://ic.media.mit.edu/.

9 https://gopro.com/en/us/.

10 https://indcor.eu/activities/task-forces/indcor-task-forces/.