Immersive virtual reality (VR) for digital media making: transmediation is key

ABSTRACT

The rapid evolution of virtual reality (VR) technologies and their adoption for learning opens up new possibilities for shifting semiotic content across modes, with underexplored scope for transmediating content in visual, haptic, and auditory ways in immersive media literacy practices. This research investigated users’ creative digital designing involving a popular, three-dimensional virtual painting program with upper elementary students who used a VR head mounted display and sensors. The analysis attended to how students transmediated the same story across written, verbal, and virtual painting modes, tracing key themes of the students’ virtual experience: (i) immersion and three-dimensionality, (ii) subjective presence, (iii) sensory illusion, and (iv) interactivity with motion tracking. Students reassembled and shifted narrative content, sometimes seamlessly, while experiencing ambiguity and complexity about three-dimensional representation in an immersive world. Producing stories across modes invited adaptation and generative thinking to overcome the lack of equivalence between writing, drawing, and virtual painting modes.

KEYWORDS:

• Immersion
• digital media
• multimodal
• transmediation
• virtual reality

Introduction

The digital and multimedia environments have profoundly changed the way we communicate in recent decades, with the convergence of technology and media, and the transformation of everyday literacy practices (Friend and Mills 2021). Recently, virtual reality (VR) technologies have rapidly become more accessible in homes and education, with different meaning-making potentials and affordances for transmediation in textual practices that include immersive storytelling, interactive reading, virtual painting, using three-dimensional models, and other emerging forms of sign-making (Dooley et al. 2020; Mikelli and Dawkins 2020; Rose 2018). Transmediation is a vital learning and knowledge generation process that is of global relevance in multimedia-based digital environments, which involves translating knowledge across sign-making systems (Siegel 2006).

The current research analyses transmediation in students’ digital media making through immersive VR environments that use a head-mounted display. The process of transmediation has emerged as salient across a range of knowledge domains when students interpret or represent concepts using varied modes of representation, such as through written and spoken words, drawings, painting, mathematics, dancing, music, and other sign-making systems (Suhor 1984). This central process of knowledge generation has been observed as essential, even among very young children when they express essentially the same content through talk, role play, drawings, and making marks on the page, or when students respond to literature (Short, Kauffman, and Khann 2000).

It has been observed that an exact translation of the same concept from one mode to another is not typical, or perhaps even possible, because modes or sign-systems are shaped from different forms of cultural work, different technologies, and often do not share the same materiality (Bezemer and Kress 2008). Since a digital shift in literacy studies, transmediation has been observed in digital media text creation tasks, such as when composing a film about a book, or when making a digital comic from a written one (Mills 2011). New ‘semiotic technologies’ have opened up different realms of meaning-making to children and young people (Lim and Toh 2020).

The present research examined immersive VR technologies with 44 upper elementary students (ages 10–12 years) to gain insight into the role of transmediation as students shifted similar meanings across modes, from written stories with drawings, to spoken retellings, and to a novel mode of virtual painting using Google Tilt Brush wearing a head-mounted display and with motion sensors. The analysis attended to the enabling and constraining features of the virtual mode for representing their ideas, and traced the key points of tension where students were required to adapt their intentions for the design in response to the lack of equivalence between modes.

What are VR technologies and how do they support learning?

Virtual reality technologies provide a synthetic or computer-simulated environment for user-immersion involving the use of a head-mounted display (HMD) for stereoscopic vision, and motion-tracking controls for haptic-feedback in the virtual environment (Velev and Zlateva 2017). These technologies have been shown to facilitate immediate and direct learning (Rasheed, Onkar, and Narula 2015), support memory (Radianti et al. 2020), and increase learner attention and motivation (Metzinger 2018). Likewise, VR has advantages for supporting understanding of visual and spatial information (Jensen and Konradsen 2018), and enabling better learner decision-making in simulated environments (Rasheed, Onkar, and Narula 2015), while also dependant on effective pedagogy (Fowler, 2015). Recent virtual technologies can now be used for collaborating (Huang, Rauch, and Liaw 2010), viewing immersive news (Steed et al. 2018), and storytelling, socialising, and learning through play (Marsh and Yamada-Rice 2018). Since the advent of full-immersion VR with HMD, screen-based virtual computing is not considered virtual reality when it does not meet the criterion of full immersion, with the external or physical world blocked from view (Jensen and Konradsen 2018).

Virtual technologies can provide access to learning scenarios that would be difficult to enact in classrooms, such as the immersive exploration of different geographies and architecture, or viewing simulations of phenomena in real time (Huang, Rauch, and Liaw 2010). Importantly, VR technologies can provide interactive and experiential learning environments that support abstract thinking and communication (Fernandez 2017). The increased affordability and accessibility of VR technology is predicted to see an exponential rise in classroom use for learning across all levels of education, from early childhood to higher education (Hussein and Nätterdal 2015).

Why is transmediation important to learning with VR?

Transmediation occurs whenever a learner maps content from one expression plane, symbolic system, or communicative mode onto another (McCormick 2011), such as using dialogue to explain ideas read in a book or writing a paragraph to explain a diagram. Transmediation was first conceptualised by Suhor (1984), who noted the unique organisational principles of each representational mode or sign-making system (e.g., written words, drawings). The concept was extended by Siegel (2006) who explained that transmediation is complex because there is no pre-existing code or ready-made link for representing the interpretant of one sign system into another. Each sign-system or mode has differing features that enable and constrain the way that ideas are conceptualised, symbolised, represented, and remembered (Forman 1994).

Transmediation has relevance for understanding the learning potentials of virtual reality technologies. This is because VR environments are multimodal, combining two or more modes or symbol systems, such as three-dimensional imagery, music, written words, mathematics, kinesics, proxemics, and gesture (Mills and Dooley 2019). VR environments involve more sophisticated and complex multimodal user-interactivity than many other technologies, which is related to the specialised features of three-dimensional and immersive VR environments.

While educational research has evaluated VR applications from the perspective of widely known learning theories, such as cognitivism, constructivism, and connectivism (see Radianti et al. 2020), the role of transmediation has not been researched in the context of virtual reality learning (Mills 2019). VR with HMD provides a novel platform for translating ideas from conventional modes, such as listening, speaking, viewing, reading, and writing, to immersive, three-dimensional media representations (Dooley et al. 2020). Because the process of transmediation involves mapping meaning across non-analogous expression planes or symbol systems, there may be generative possibilities for learners to adapt their ideas and representations in VR contexts.

Materials and methods

The purpose of the study was to understand the potentials and constraints of virtual reality for translating ideas from conventional modes, such as writing and drawing, to an immersive virtual painting mode, including the moments of tension in which students had to invent new ways to represent their ideas in response to the lack of equivalence across modes.

Three upper elementary teachers and the research team planned the learning tasks together, aligned to outcomes in the Australian Curriculum English (e.g., use software to create texts), Technology (e.g., communicate ideas applying technical protocols), and the Arts (e.g., develop skills with media technologies to create texts; ACARA 2021). The students were introduced to Greek mythology including Icarus and Daedalus, Daedelus’ Maze (Labyrinth) and the Trojan war. A range of media were used over several lessons (e.g., books, presentations, internet). The students wrote story retellings of the Greek myths and illustrated these using pencils on paper. Learners translated these ideas from their written and illustrated story retellings to the mode of virtual painting using Google Tilt Brush and the HTC Vive system (see Figure 1).

Figure 1. Natalie using HMD.

Research question

In the context of this upper elementary school program, the research specifically aimed to answer the key question:

What features of the VR learning experience are enabling or constraining when transmediating content from writing and speaking, to virtual reality painting?

Transmediation plays a significant role in many contexts of learning, communication, and composition, because information is often represented and recontextualised using different modes and media (Mills 2011).

Site and participant description

The 44 student participants in this research were from an independent, elementary school in Queensland, Australia aged between 10 and 12 years (Year 5). The cohort was culturally diverse, with 31% of the local population born overseas, and 69% born in Australia. The region had several markers of socioeconomic disadvantage, such as low tertiary education attainment and unemployment above the national average (ABS 2016). The caregiver consent rate was higher than 90% for this research.

Data collection

The students were shown a YouTube video about how to use the HTC Vive controls and virtual paintbrushes. This occurred prior to virtually painting a scene inspired by their written story retellings of a Greek myth. Students observed peers creating their virtual design to gain familiarity with the virtual painting mode. The majority indicated in interviews that they had encountered other VR technologies previously in recreational sites with friends, with VR becoming prominent in students’ everyday media worlds.

The research team conducted the VR activities over a five-week period, with a high-end personal computer supporting the HTC Vive hardware, which was funded and set up by the research team. The VR tasks were conducted by university researchers and two assistants, recording one participant at a time. Data collection involved the following multimodal and digital data about the learning experiences for 44 participants:

1. Students’ story writing – retellings of Greek myths with illustrations.

2. Interviews – student pre-task and think-aloud (during the task) interviews.

3. Video observations of each student using the VR headset and sensors (Click to view sample video: https://drive.google.com/file/d/1bhqjdnUkf-lSDD-LDOfZ1UR7ijiN3sgT/view)

4. Continuous screencasts using OBS Studio free online software (View sample, audio removed: Evie: https://drive.google.com/file/d/1XGkjKg6rrn_0nIW4vsRKMaNvXY7EpVlY/view; Zara: https://drive.google.com/file/d/18Vec8yVeFCs0rYFFuQHYcqV8se9wCH0P/view).

5. Videos of the virtual paintings filmed within Google Tilt brush.

Each student engaged in video-recorded pre-interviews before using the VR equipment and answered semi-structured, think-aloud questions to verbalise their thoughts during the virtual designing task (see Figure 2).

Figure 2. Sample Think-aloud.

Think-aloud interviews are useful for understanding learning and thought processes (Hevey 2020). The think-aloud protocol addressed questions designed to elucidate the students’ thoughts on the enabling and constraining features of the virtual mode, such as: What is easy and what is difficult about representing your story as a three-dimensional digital format? What did the VR allow or constrain you to do compared to your story writing?

Data collection involved 21 continuous hours of classroom video observation as the participants engaged in virtual painting while wearing the head-mounted display (15 minutes per student), in addition to preparing the virtual environment the morning or day prior to the research. Screencasts comprising 21 hours of recording duration were simultaneously made using OBS Studio software, capturing the virtual painting process and the students’ haptic activity using the virtual control panels, as well as student audio from the head-mounted display. Haptics here refers to human interaction with the external environment through touch or movement of the hands (Minogue and Gail Jones 2006). After the fieldwork was completed the 44 auto-recordings of the virtual paintings were replayed by the researchers while wearing the head-mounted display to convert the auto recordings into a widely accessible file format (mp4) for viewing and analysis on a computer.

Video recording was necessary to generate an accurate record that could be replayed multiple times and analysed moment-by-moment to attend to the process of transmediation, a multimodal process too complex for researchers to analyse from field notes (Loizos 2008). A multimodal record was then created from screenshots of the video images and transcribed speech, screencast, and virtual paintings. This enabled the researcher to compare the key similarities, differences, and moments of anomaly for the learners when translating ideas across modes, attending to both the creative processes and outcomes of the learning task. Ethical clearance was approved by the university human research ethics committee following Australian guidelines for conducting research (ARC 2018). Students and their caregivers provided voluntary, informed, and written consent to participate, including for the sharing of video recordings using pseudonyms of students wearing the VR HMD.

Data analysis

Multimodal interaction analysis was chosen for the data analysis because understanding the process of transmediation required synthesising, comparing, and evaluating information across multiple modes and media (Norris 2004). The full record of pre-interview and think-aloud interviews for the 44 participants was transcribed and initially coded using NVivo 1.0 to identify the frequently observed enabling and constraining features of virtual painting for shifting meanings across modes. The researchers also used MS Excel to compile a condensed, systematic record describing and summarising the complete set of written story thumbnails, virtual art, and student dialogue to compare the modes for each student. Figure 3 (Part A and B) is a simplified sample of synchronised data for a Participant subset.

Figure 3. Part A: Drawing, VR painting, and think-aloud. Part B: Drawing, VR painting, and think-aloud.

A multimodal analysis was performed by comparing and analysing the data sets: (a) the pre-interview and think-aloud dialogue: (a) the written stories, (b) virtual paintings, (c) at-screen video, and (d) OBS Studio screencast of the design process.

Aligned to the research question, the multimodal interaction analysis attended to the key enabling and constraining features of virtual painting for transmediating content to the virtual mode. The researchers identified key moments across the data in which the students adapted their ideas to create condensed or expanded meanings, rather than simply reiterating the narrative, given the lack of equivalence between modes. The following main themes emerged as significant in translating stories to the virtual painting mode, which were refined using relevant terms from the literature: (i) immersion and three-dimensionality, (ii) students’ subjective experience of presence, (iii) students’ perceptions of sensory illusion, and (iv) interactivity with motion tracking. These features of the VR experience were both enabling and limiting for transmediation, while translating ideas to the virtual environment created the need for adaption and generative thinking to overcome the lack of equivalence across modes.

Results: VR experiences for transmediation

The findings are presented in four parts that follow the main themes from the coded and analysed data, elucidating the enabling and constraining dimensions of the virtual experience for shifting meanings from conventional writing and drawing to virtual painting. The results attend to instances where the translation of meaning created a lack of equivalence between modes, inviting the learners to create new meanings or adapt their intentions for their designs.

Immersion and three-dimensionality

The majority of the participants experienced the immersive, three-dimensionality of virtual painting as supportive for transmediating meanings across modes. Immersion is the way in which a three-dimensional, simulated environment surrounds the user and supports the illusion of being real, while the physical world is blocked from view (Jensen and Konradsen 2018). In the pencil drawing context, most students depicted a Greek mythical scene using two-dimensional elevation drawings or profiles, which contrasted the immersive three-dimensionality of their representations in the virtual mode. For example, when using pencils on paper, Emily depicted the gate in the city wall of Troy, the location of the Trojan war, from two-dimensional elevation view (Figure 4, left). In contrast, when painting virtually, she optimised the immersive and three-dimension possibilities to create a city able to be explored on foot by virtually moving through the depicted city gate (Figure 4, right).

Figure 4. Emily’s 2D drawing and 3D painting.

Interestingly, Emily’s written story about the city of Troy adopted a first-person perspective as a farmer called Sarah growing vegetables and surveying the green meadows, paralleling the subjective viewpoint of her virtual artwork that was produced from the view of a Greek inhabitant of the city (Figure 5).

Figure 5. Emily’s written Greek myth.

Further, Emily’s ideas were extended and adapted to suit the artists’ first-person perspective and physical immersion in the virtual painting environment, such as adding fiery special effects to intimidate the enemy, described in her think-aloud (see Figure 6).

Figure 6. Emily’s think-aloud.

When asked directly about enabling features of the VR environment for shifting the Greek story content, students alluded to the three-dimensional, immersive features of their virtual painting. For example, when asked: ‘What’s different about the VR artwork compared to your story,’ Isabella explained that in the three-dimensional, virtual space: ‘I can show what’s inside the Trojan horse’ (see Figure 7).

Figure 7. Isabella’s 3D Trojan horse.

Similarly, other students alluded to the three-dimensional immersive environment. Alexander’s virtual artwork was about the Greek soldiers fighting, and descending from of the huge, wooden Trojan horse. Alexander described virtual painting:

You have all this space – all around you – instead of just one small piece of paper. I like how you can do different options to paint items, different effects like fire. You can do a fiery thing over there, and something else over there. It’s amazing all around.

Likewise, drawing attention to the dynamic representational resources associated with the immersivity of the virtual painting, James, painting Icarus and the sun, noted: ‘It made everything come off the page. The written story and drawing were grey, whereas the 3D art was in colour.’ James also noted that the special effects were not static, but moving, flashing, and dynamic: ‘It wouldn’t like, just stay there and not move. If you added the wave effect it would look like it was moving like a wave.’

Other students depicted their Greek myth on paper using aerial perspective, which often creates an impersonal and objective view as a distant observer. For example, Hannah retold the story of Daedalus and Icarus, including a pencil drawing of the maze and tower. She used an aerial perspective of the maze, juxtaposed with a profile view of the tower where Daedalus and Icarus were imprisoned (see Figure 8, left image). This contrasted the three-dimensionality of her virtual painting, where she created the scene as if personally located inside the maze and tower. Her virtual painting had an emphasis on still life textures, creating a sense of ambiance inside a dark, historic, and immersive space: ‘I want it to look kind of old … I wanted to use oil paint because it kind of almost looks like wood, and the wood looks old’ (see Figure 8, right image).

Figure 8. Hannah’s 2D drawing and 3D painting.

Hannah noted that the virtual technology facilitated the inclusion of textural features in her artwork and other elaborated details in her mental and virtual construction of the events:

I think I can add better detail in VR, I don’t think I was able to have this texture in the drawing. It’s adding more detail and gives the story a kind of different message.

At the same time the immersive three-dimensionality of the virtual painting mode posed some constraints for a group of students who struggled to paint recognisable forms in three-dimensions, but who drew detailed two-dimensional drawings on paper. For example, Ava’s two-dimensional elevation drawing of the Trojan War was clearly drawn and included technical details (see Figure 9).

Figure 9. Ava’s 2D drawings.

However, in the three-dimensional environment Ava struggled to portray the Trojan Horse with the equivalent skill and clarity. She made use of the ‘undo’ function multiple times to erase each attempt to draw the body and legs of the horse, sometimes because of unwanted virtual paint thickness or texture. She was observed repeatedly scrolling through the tools options to locate the right effects, colours, line thicknesses, and textures. After almost nine minutes of recreating the Trojan Horse, Ava erased much of the painting, leaving all but the outline of the oval-shaped head and ears. After ten minutes of virtual painting Ava was still struggling to paint a sufficiently recognisable horse using a simple two-dimensional profile view with few textures or details (see Figure 10).

Figure 10. Ava’s 2D drawing and 3D virtual painting.

Others such as Eli, who effectively shifted meanings across writing, drawing, and virtual painting, similarly encountered certain constraints at times. Eli spent considerable time searching through the virtual controls looking for ready-to-use three-dimensional shapes, while making negative evaluations of the tools: ‘No, it’s too clear. No, it’s just flat.’

Many students, including those who were observed to work more skilfully with three-dimensional representation acknowledged the challenges, as Sienna noted: ‘It’s very complicated compared to the paper one.’ Sienna’s virtual representation demonstrated strategic use of three-dimensionality, depicting the outline of a complex maze with dead ends and passageways using isometric projection often used by engineers and technical illustrators, combined with iconic Greek column props (Figure 11).

Figure 11. Sienna’s 3D isometric Greek maze.

When asked if there were any constraints of the virtual painting mode, Sienna concluded, ‘It’s hard!’ One student noted that he had chosen to draw two-dimensional profile drawings in the virtual environment for greater efficiency given the time constraints. The majority of students combined both two and three-dimensional forms or layouts in their virtual designs (n = 34), with the remainder working only with three-dimensional forms (n = 10), and no student using only two-dimensional techniques. To illustrate the combined use of two and three-dimensional forms, Natalie’s Trojan Horse in the virtual environment assumed the perspective of a distant observer and avoided the challenges of creating a fully immersive scene: ‘Mine is The Trojan Horse, but it’s a different way of doing it [to other students]. Instead of actually being in Troy, it’s at a distance from it. There’s homes everywhere, and you can just see that at a distance.’ The three-dimensional and immersive features of the virtual painting mode were underutilised in this captioned, elevation painting, though the images were placed around the artist to create some depth (see Figure 12).

Figure 12. Natalie’s Trojan Horse.

Students were able to include a larger number of elements and details in their written stories and two-dimensional drawings than in the three-dimensional, virtual scene. For example, Jackson’s written story provided sensorial details of Troy through descriptive writing, such as the smell of smoke from chimneys, the sound of wolves howling, the barren ground, and the darkness of the forest surrounding the castle. He also provided the mathematical dimensions of the castle walls, described the shape of the wall blocks and spikes, the moonless starry sky, the size of the waves on the ocean, the number of boats sailing into land, and the sounds of a party inside the castle walls. However, when Jackson began to use the VR tools, he identified the need to reduce the number of elements for a three-dimensional world: ‘I think I’m going to have to just go with the castle and the boats.’

At the same time, Jackson acknowledged an ease in the virtual mode to convey his mental imagery fluidly and intuitively: ‘Instead of thinking: What would that look like? Instead of having to think how to describe it, I just knew it already. I could just do it.’ Jackson created an immersive night scene from the perspective of a Greek soldier standing between a vivid blue sea of boats and the towering walls of the city of Troy. The immersive, three-dimensional virtual environment provided novel features for enlivening the students’ narratives, which the majority of the students found enabling, as Lily described: ‘Because you can actually see it, and turn around in it, and then see other things.’

Presence and subjective perspective

The majority of participants experienced a sense of presence in the virtual mode – of being there – when shifting their ideas to the virtual painting mode. Presence is a subjective state of the user in VR environments and is not a feature of the technology itself (Slater et al. 2009). It denotes the subjective experience of being located in a virtual place, rather than viewing as someone observing externally (Kavanagh et al. 2017). Presence is experienced on a spectrum that depends on the quality of the immersion and degree of multisensory feedback (Light 2019).

A number of students experienced a sense of presence in VR. For example, Hunter’s artwork depicted Icarus flying too close to the sun, with Hunter’s own body taking the place of Icarus in the virtual scene. Hunter explained before painting his scene he wanted to depict what Icarus experienced as he flew above the world:

It’s going to be from his eyes … he’s not going to be shown in it. It feels like he knows what it’s like to be a Greek god. So, he goes up too high and the sun burns him, and he goes into the ocean and drowns.

In the virtual space Hunter rotated the controls around his body making a 360-degree circle to form an aerial view of the maze (see Figure 13).

Figure 13. Hunter as Icarus.

At one point, Hunter called out: ‘It kind of feels like I’m falling.’ Hunter experienced a sense of presence as Icarus in the scene and continued: ‘I like it!.’ He then created wings which reached out from the sides of his own body. Icarus was not represented, because Hunter inhabited the virtual space as Icarus. The virtual art in Figure 14 shows the glowing wings of Icarus, the space between the wings inhabited by Hunter who stood close to the sun, while viewing the maze and shimmering water below.

Figure 14. Hunter’s VR painting

In another clear example of presence used in the virtual environment Jackson described himself as a Greek soldier approaching Troy: ‘I’m a Greek soldier who’s just come in at night, I’ve just walked up the bank, and I could see the castle and its background, and all the men coming behind me.’ Jackson represented water with boats, and opposite, a castle wall, locating himself on the shore between the ocean and the castle (see Figure 15).

Figure 15. Jackson’s VR painting – three views.

Recognising presence, which current VR spaces typically project (Metzinger 2018), Madison alluded to how the virtual technology allowed her to produce a greater ‘sense of it’s here’ and ‘it’s real.’ Madison noted that she could ‘actually visualise’ the scene. She moved around the three-dimensional, virtual space creating the designs around herself, valuing the ability to move inside her immersive virtual painting: ‘You could see what was happening around you. Like you could turn around and make stuff, instead of it just being in front of you. It’s like you’re in a gallery of photos put together into a shape around you.’ Below are two snapshots of her virtual painting, including a longer, composite image (Figure 16).

Figure 16. Madison’s VR painting – two views.

In this way, the virtual painting mode permitted a degree of presence and phenomenological realism, which previous studies have described as focusing the user’s attention on the subject matter (Metzinger 2018). As demonstrated in some of these examples, a sense of presence in the virtual environment supported students’ vicarious participation as actors in the Greek narratives. The virtual environment afforded by the HMD similarly enveloped their vision, contributing to the sense of presence in the process of transmediation, without visual distraction from the real world.

Sensory illusion

The role of sensory illusion in transmediating meanings from conventional to virtual modes was observed to both enable and constrain the learners at times, with more frequent occurrences of constraints. Sensory illusion in VR environments can take several forms that each contribute to an overall sense that virtual actions, places, and bodies are real. Forms of sensory illusion are sometimes seen as components or features of a virtual experience that contribute to the user’s overall subjective sense of presence (Steed et al. 2018). Embodiment illusion and body ownership in virtual environments pertain to the sense that a virtual body or body part is their own and is a source of the experienced sensations (Tsakiris 2010). In relation to a feeling of self-location, place illusion is the sense of being transported to a new location (Gonzalez-Franco and Lanier 2017), a perception that might be experienced, such as using a teleport function in a virtual world.

When asked ‘What did the digital technologies allow you to do that you couldn’t do in your story?’ William identified the value of being able to teleport. He explained: ‘I was able to paint so that I could go out of the horse. And I could see there was like, a big horse shape. Then I could walk back into it, and then it would have all sorts of stuff.’ Likewise, Lily used the teleport tool to move around the artwork, both inside and outside her Trojan horse, making it possible to view the three-dimensional objects from various angles.

However, the sensory illusion of embodiment in the virtual environment was not always perfect or complete. When Sienna shifted her story across modes, she observed that the hand controls were depicted in a somewhat disembodied way in the virtual view, since there was no corresponding illusion of the users’ hands, limbs or other virtual body, while the user’s body is blocked from view. During the think-aloud interview Sienna commented: ‘It’s really weird because when I touch it, my hands aren’t shown there.’

Sienna also found it confusing to manage the sensory illusion of place because the physical world was blocked from view. As she moved further toward the periphery of the virtual space, she commented: ‘Ah, I don’t know how much I’ve moved here.’ Sienna became disoriented about her physical location in the classroom setting, actively searching for locational clues, such as the classroom wall, using outstretched hands, indicating that the sensory illusion of being in another world was somewhat incomplete. Others, such as Evie, experienced the sensory illusion of place as generative for shifting meanings across modes. Evie’s two-dimensional drawing using pencil on paper was based on the story of Icarus, providing an aerial view of the minotaur in a maze (see Figure 17).

Figure 17. Evie’s drawing and VR maze.

Evie became absorbed in the illusion of place as she created the mythical Greek maze in a three-dimensional world: ‘I’m going to create the bull in the maze because I thought it would be cool to see it from a different point of view, instead of just trying to imagine it.’ Evie moved across the virtual space as she stepped out the maze, which became increasingly real: ‘I’m doing the maze green because it’s plants and stuff. This is the brown door.’ Evie appeared to relish materialising the minotaur in 3D: ‘It’s the bull horns. Yes, that’s good. They’re like fangs. It’s supposed to be evil. It’s a very evil bull that kills children.’ Evie became fully captivated by designing in the virtual mode, immersed in the sensory illusion of being located inside the maze.

At other times, the students reflected spontaneously about the sensory illusion in the virtual space. For example, Jackson described how the virtual environment was so real that he could ‘feel’ it. However, he also noted that the sensory illusion of place was disrupted by an inability to teleport, perhaps limiting his ability to create greater distant objects in his virtual scenery. When asked: ‘What did the digital technology stop you from doing that you could do in your story?’ Jackson replied: ‘I guess that I couldn’t make it to a distance is what I’m trying to say.’

Some students found the sensory illusion of scale somewhat difficult to optimise in the virtual painting experience, despite being shown how to shrink and enlarge created objects using the tools. For example, Charlie’s written story included a ‘king’s castle wall’ that was ‘the size of our classroom.’ Charlie found that, overall, the interactivity of the virtual environment allowed him to quickly create his virtual artwork (see Figure 18).

Figure 18. Charlie’s VR castle.

However, Charlie identified that it was also ‘harder to understand size. Like, if I do this ever again, I may do it much bigger than this, and do it in a better size.’ For example, as he became familiar with the sense of virtual scale, he repeatedly enlarged his artwork. He stepped through the virtual space, reaching high and low, creating an extensive city wall. Charlie was not alone in searching for a better way to translate the scene to a virtual scale, particularly for creating the illusion of being surrounded by enormous objects or vast landscape.

The auditory environment did not play a very important role in the virtual painting mode, with the software offering only sound effects to the rhythms and visual effects of the virtual paint brush. William, who used the VR tools to create an interior environment of the Trojan horse, noted that the virtual painting lacked realistic audio elements to enhance sensory illusion of being situated in the virtual place: ‘When I was painting I couldn’t really hear much, but in the story, if it had noise I would be able to hear all of soldiers in there with me and probably the Greek boats coming back from sea.’ Previous research has also suggested that being able to converse with others in the real world is sometimes experienced as disruptive in rendering the sensory illusion of the virtual world due to the incomplete exclusion of the physical world (Fernandez 2017).

Interactivity using motion tracking

Interactivity using motion tracking was observed to be both enabling and constraining for transmediating content to the virtual painting mode. Digitising user movement is a key component that responds to the user’s sensory input to support movement-based interactivity. Motion tracking enables users to walk around in the virtual space, interact with virtual objects, and create virtual designs with the controls, sensing head movement and gestures, while updating the user’s view correspondingly in the virtual world (Velev and Zlateva 2017).

At times, haptic interactivity supported the students’ intuitive designing, becoming an enabler for shifting meanings to the virtual painting mode. For example, Ethan was observed making a series of extended circular and lateral arm movements, decreasing in size, to create a bright yellow sun. Figure 19 shows his virtual painting on screen corresponding with his haptic movement and supported by his head and body position.

Figure 19. Ethan’s haptics to create the sun.

At other times the students were observed making much larger body movements, with interactivity involving locomotion throughout the virtual space, along with guiding movements of the head and arms. For example, in Figure 20 below, the motion sensors of the hand controls allow Emily to step, turn, crouch low to the ground, and reach high into the air to create her concept of the wall of Troy, demonstrating a magnitude of gross-motor movements and interactivity that is not typically paralleled in writing and drawing.

Figure 20. Emily’s gross-motor interactivity.

Optimising interactivity in VR, Charlie created a three-dimensional design in which he surrounded himself by the wall of Troy using the Papier Mache tool. He outlined the wall with scaffolding before returning to attach vertical wooden slats, relocating his virtual brush, set on a large brush-width setting, quickly and dextrously. He explained that the tools were ‘allowing me to do different types of designs. It’s really easy, and you can cover up spaces with just little movements, because you have more control over it’ (see Figure 21).

Figure 21. Charlie’s layering of virtual paint.

While students most often optimised haptic and locomotive interactivity in the virtual mode, there were some limitations that resulted in students needing to adapt their written stories to overcome the constraints of interactivity. These constraints were associated with difficulties accurately representing fine details when moving within an unstructured virtual or immaterial void. For example, Madison was able to create a Trojan scene around herself, but she felt VR prevented her from ‘creating really intricate things, intricate buildings.’ Madison’s written story that described the City of Troy was characterised by much greater detail compared to her story, a short excerpt shown below:

It was a cold, dreary morning. Through the open gates I could see an armada docking at the beach, their soldiers forever getting closer. As I walked down the rough streets, I saw the blocky horses, made from clay and rocks. The palace loomed through the fog, blocking my vision of the humongous mountains.

Madison’s written story portrayed temperature (cold), mood (dreary), time of day (morning), historical detail (armada, blocky horses made from clay and rocks), and conditions of visibility (loomed through the fog, blocking my vision), among other vivid descriptions of place and action later in the story. Yet when reflecting on the virtual mode, she alluded to the need for greater precision and control to represent detailed virtual objects through haptic interactivity. This is confirmed by comparing the number and quality of her written descriptions to the minimalistic detail in her virtual artwork, which was schematic and simplified. This pattern of simplification and abstraction of ideas in the virtual painting context was a repeated theme, observed when comparing the students’ written stories to their corresponding virtual painting, and by the students’ self-reported reflections in the interviews.

This simplification of ideas in the virtual mode was often associated with the precision of the motion-tracking interactivity. For example, William found it challenging to locate his virtual brush to connect lines and surfaces:

I found the 3D hard because if you painted something, and then you wanted to paint something on top, you had to lift the brush up slightly. I found it hard to layer it perfectly without it getting a little bit messy.

Emily similarly identified the difficulty of matching or connecting her paintbrush to previously drawn forms in the virtual space in the absence of a tangible surface, though she employed an outlining technique to guide her haptic interaction. For instance, she used a vertical line or support post at each end of each section of city wall, and then filled the space with multiple horizontal strokes.

In another illustrative case, Mason created a vivid, bright depiction from the Trojan war. Prior to using the digital technologies Mason had planned to create a ‘threatening’ action scene from the Trojan war, including ‘a castle and some water, and boats around the castle and spears on the top, so they can’t get in.’ Mason anticipated difficulties translating his original story content to the virtual mode because ‘you can’t make everything the same in your mind and virtual reality.’ After using the digital technology, Mason reflected on the virtual mode:

It’s harder to paint it in VR. You think about something where the lines are all straight, but it’s really hard to get the lines straight. I felt weird because, like, you’re painting in mid-air … It’s weird!

The immateriality or lack of tangible substance of mid-air painting in the virtual world contrasts painting in the physical world, because an artist typically receives sensations and information through haptics as their fingers, paint brush, or other tool follows the contours of a physical surface. A painter of physical objects can modulate the pressure applied to the brush, and process information gained through haptics or physical touch about the properties of the surface painted, such as hardness, distance, shape, size, and texture (Minogue and Gail Jones 2006). In contrast, virtual painting was characterised by a lack of corporeality, coupled with the absence of virtual sensations or haptically-perceived information to create a sensory illusion of painting against a physical surface.

However, Mason articulated how he responded and adapted his intentions for the design:

But then I just went with it, and put other things like windows and rainbows … It was a lot harder, but then I thought, ‘I’m not going to erase everything.’ So I just kept going. I did add some other visual effects that were not mentioned in my story, like grass, and rainbows, and stars. (see Figure 22)

Figure 22. Mason’s VR Trojan War.

There were some elements in Mason’s virtual design that were consistent with the original concepts in his written story, such as the inclusion of approaching boats, fire, the city, and the night-time setting, but in a simplified form. At the same time, the virtual mode permitted Mason to optimise the radical haptic interactivity, quickly surrounding himself with objects in a curved landscape that offered some depth. In sum, the interactive nature of VR, which responds to user movement in a full immersion environment, provided both opportunities and constraints for shifting ideas from writing and drawing to the virtual mode. The immersive and large-body physical interactivity provided a novel element, with gaps and areas of improvement for representing fine details, given the absence of tangible surfaces, or some artificial stimuli to give the user a sense of pressing against a corporeal surface.

Discussion: potentials of virtual reality for transmediation

This paper has examined the enabling and constraining features of the virtual painting mode for students in immersive, three-dimensional, simulated environments, who were engaged in mapping content from the widely used expression planes of writing and drawing. Students reassembled and transmediated narrative content, sometimes seamlessly, while at other times, they drew attention to the ambiguity and complexity of three-dimensional representation in an immersive world.

The virtual environment offered a new and immersive simulated world to translate ideas from Greek mythology that would have been impossible to experience first-hand in the real world, while offering enhanced visual and haptic-based ways to shift ideas. Learners were supported to convey novel subjective positionings in a three-dimensional, simulated environment, while requiring users to gain familiarity with three-dimensional virtual affordances that are not equivalent to conventional writing and drawing, but which required users to generatively invent and adapt ways to simulate concepts through meaningful and readily interpreted virtual symbols.

A novel and advanced feature of the VR environment for transmediation was immersion and three-dimensionality, which was largely supportive for transmediating ideas, enabling students to transform their simple two-dimensional, elevation drawings into dynamic depictions of three-dimensional Greek mythical worlds of detail and complexity. This is aligned with previous VR research where greater immersion led to an increased interest in the depicted subject-matter (Metzinger 2018).

A sense of presence in VR was predominantly enabling because students perceived that they were inside their texts, walking around the city of Troy, and subjectively experiencing the Greek worlds with heightened imagination. Experiencing presence occurs when the user can actively modify their surroundings, with successful substitutions of real sensory data with virtually generated sensory data (Slater et al. 2009). Relatedly, sensory illusion was found to be both constraining and enabling, with the lapses or gaps more often noted, consistent with other research where the virtual representation creates disassociations between the user’s sense of body ownership (Steed et al. 2018).

Bodily and haptic interactivity was heightened in the virtual mode, which differed to drawing and writing, and which was both constraining and enabling for students. These movements made it easy to paint large areas quickly, and to make structures as tall as the students’ reach. But students also had difficulty accurately representing fine details and connecting lines when moving in an unstructured, immaterial void, despite sound ‘visual-tactile synchrony’ in the virtual painting experience (Steed et al. 2018, 5).

While the results include both strengths and weaknesses of the virtual experiences, these features contributed to an original, simulated learning experience that provided new potentials for transmediation or shifting of knowledge across multiple modes. The findings address a widening gap in media research that has focused on virtual technologies that are less immersive, such avatar-based applications in virtual worlds (O’Connor and Domingo 2017), web-based VR (Huang, Rauch, and Liaw 2020), or limited forms of presence using an internet browser (Light 2019).

Future research will be needed to examine the potential and constraints for transmediation in multimodal design across a growing range of VR applications for educational outcomes, and across other sign-making systems, such as through engaging in interactive VR documentary films, or developing facility with written words, oral language, mathematics, music, sign-language, gestures, in virtual and non-virtual environments.

Conclusion

In sum, since the advance of wearable, head-mounted displays, virtual reality technologies have become readily accessible, immersive multimedia that simulate the user’s physical presence in an artificial, three-dimensional environment (Velev and Zlateva 2017). The rapid evolution of VR technologies and their adoption for learning opens up new possibilities for shifting semiotic content across modes, with underexplored scope for translating narrative content in visual, haptic, and auditory ways in media design.

The enabling and constraining features of VR environments to support specific modes and ways of thinking are key because each symbol system influences both what and how knowledge is communicated. The findings of this paper are significant when education systems are critiqued for privileging spoken and written modes (McCormick 2011). The findings highlight key advantages and constraints of VR technologies for shifting meanings from conventional modes of drawing and writing to new forms of virtual immateriality.

Virtual reality applications, like all technologies, can be used for ethical or unethical purposes, or be used critically or uncritically, to serve the interests of dominant or marginalised social groups. For example, VR (360 degree) non-fiction films or documentaries can generate powerful empathy with the documentary subject. This ability to elicit empathy in VR film could be misused to distort truth (Rose 2018). Similarly, experiencing the world of the subject could lead to the moral risk of improper distance (Nash 2018). In contrast, the ethical use of 360-degree VR film making has shown the transformative potentials for youth to tell their stories, channelling the immersive potential of VR to create empathy for the marginalised (Mikelli and Dawkins 2020). Historically, moral questions that are raised about VR were similarly asked about older media which play with the real – books, photography, cinema, radio, and television – which can be used ethically or unethically (Horsfield 2003).

Virtual reality technologies can support a plethora of pedagogies, with clear advantages for experiential and situated learning, and applied knowledge (Radianti et al. 2020). However, this research has shown that VR technologies can be used to support the translation of knowledge across sign-making systems. Even when a task seemingly calls for a simple reproduction of declarative knowledge learned through reading, listening, drawing, and writing, the process of shifting this knowledge into a new mode, such as virtual painting, is never a simple reproduction of content. Rather, the lack of equivalence between the modes invites learners to engage in a generative process of adaption and knowledge transformation (Mills 2011). This is because learners must recast knowledge via increasingly complex, multimodal, immersive, and interactive, yet physically intangible, three-dimensional expression planes that now feature in education.

Acknowledgements

We acknowledge the contribution to fieldwork by Mark Williamson (Big Picture Industries Inc.), Dr Lesley Friend (ACU), and teachers at Silkwood Primary School. We thank Professors David Howes, Jim Gee and Theo van Leeuwen for their advisory work.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was supported by an Australian Research Council (ARC) Future Fellowship awarded to the Professor Kathy A. Mills [grant number FT180100009], funded by the Australian Research Council. The views expressed herein are those of the authors and are not necessarily those of the ARC.