The translation of multimodal texts: challenges and theoretical approaches

ABSTRACT

This article introduces a thematic issue devoted to the translation of multimodal texts as well as some of the theoretical approaches used by researchers. It provides an overview of some of the most recent publications on the interaction between multimodality and translation as well as some of the challenges that both translators and researchers encounter when dealing with multimodal texts. Drawing on functional linguistics, Kress and van Leeuwen proposed a model of the analysis of ‘visual grammar’ that has been widely used in several disciplines, including Translation Studies. Some of the articles included in this thematic issue also draw on Kress and van Leeuwen’s approach, while others explore the modes of the source texts (including films, opera, museum texts, comics, and video games) by considering linguistic, cultural, and technical aspects from various theoretical standpoints.

KEYWORDS:

• Translation
• modes
• verbal
• acoustic
• visual
• multimodality

Introduction

Multimodality or ‘the use of several semiotic methods in the design of a semiotic product or design’ (Kress & van Leeuwen, 2001, p. 20) characterizes most contemporary cultural products (e.g. movies, videogames) and venues (e.g. museums, theaters). Images are an integral part of the construction of contemporary texts, whose message is shaped by the combination of images, sound, and writing/speech. These layers of meaning can be complementary but also contradictory. Thus, they need to be taken into account when language is rendered into a target language (Tuominen et al., 2019). Yet, as Kress and van Leeuwen (2006, p. 41) point out, the multimodal nature of texts has been traditionally ignored by researchers of most disciplines, including Translation Studies (Taylor, 2016), even though some authors have spoken of a multimodal turn in the discipline (Boria & Tomalin, 2020). It is indeed doubtful whether the use of different modes for the generation of meaning should be analyzed separately (Kress & van Leeuwen, 2006, p. 177) because, as Taylor claims, the interaction between the verbal elements and the visual, acoustic, kinetic and other semiotic modes are crucial to convey information (2013, p. 100). In fact, meaning can be conveyed more directly via one or more of the other communication modes than via the words themselves.

However, as mentioned, the interaction between these layers has not received the attention it deserves, probably because their analysis requires the use of a variety of theoretical approaches. In this respect, multimodality is not so much a theory as a field of application (Bezemer & Jewitt, 2010, p. 180) or enquiry, which might necessitate an eclectic combination of theoretical approaches. The interest in the analysis of multimodality, though, stems from social semiotics and the seminal work by Kress and van Leeuwen, who, in turn, drew on the tenets of Halliday’s functional linguistics. Unlike structuralists, Halliday analyzed language considering its social and communicative dimensions and introduced concepts that have been widely used in other disciplines. Of particular interest is, of course, the notion of ‘mode’. ‘Mode’ refers to a set of resources capable of making meaning (Adami, 2023). In social semiotics, ‘mode’ is preferred to ‘code’, as its boundaries are less stable and allow for more flexible approaches to study how meaning is generated.

The use of the term ‘mode’ in other disciplines has led to terminological problems (Adami, 2023; Valdeón, 2018). In fact, the word ‘multimodal’ has been applied to different concepts in Translation Studies. Of note is Snell-Hornby’s categorization of texts combining two or more modes: multimedial (or audiovisual), multimodal (involving modes of verbal and nonverbal expression such as drama and opera), multisemiotic (involving different graphic systems, such as advertising brochures) and audiomedial (or texts written to be spoken, as political speeches) (2006, pp. 84–86). According to this classification, multimodality would apply to texts written to be performed live. Although Snell-Horby’s taxonomy manages to capture differences between various text types or genres, ‘mode’, and, therefore, multimodality, remains the most-commonly used term for the various text types that the contributors to this thematic issue discuss. In addition, as Adami (2023) points out, ‘multimedial’ typically refers to the technologies used to produce multimodal texts. Thus, the radio makes use of speech and sound, while television and the internet also use images, which will also include gestures, facial expressions, and so on.

Be that as it may, the fact that the modes shaping most contemporary texts have not received much attention until recently (and often in isolated instances, Adami, 2023) is indeed surprising, given the multimodal nature of audiovisual products such as movies and television programs, but also of literary texts such as Lewis Carroll’s Alice in Wonderland and Kenneth Grahame’s The Wind in the Willows. The latter, for instance, was originally published in 1908 as plain text, but later editions included drawings by different illustrators, who left their own mark on the (reception of the) book. For instance, the drawing of the Rat and the Mole sitting in front of a fireplace by Ernest H. Shepard (Grahame, 1908/1933, p. 25) serves to complement the plot by creating an image readers will be familiar with. Illustrations by other artists, though, are likely to produce different effects on the readership, as each one held different views of Grahame’s work. In total, Hares-Stryker (2009) records almost a hundred different illustrators but little has been published on the relationship between the text by Grahame and the images by the various illustrators, who interpreted Grahame’s words in different ways. Hares-Stryker herself merely lists the names of the illustrators together with some biographical data.

And yet, illustrations can contribute to recontextualizing literary works to the extent that the original texts can be appropriated by target cultures for their own purposes. I have discussed the translations of Brevísima relación de la destrucción de las Indias, a pamphlet written by Bartolomé de las Casas for the eyes of Crown Prince Philip of Spain, which was later published, translated, and illustrated by the rivals of Spain in the colonization of the Americas (Valdeón, 2014, 2017). Most of the translations of the semi-fictional accounts written by the Dominican friar, who claimed, amongst other things, that the rivers in the island of Hispaniola were as long as the Ebrus and carried enormous amounts of gold, included drawings by Theodor De Bry in which he aimed to portray Spaniards in a very negative light. Thus, a text written to be read became a text that has been inextricably linked to a few images, which have largely replaced the written word and original intention of the author.

Audiovisual translation

Most of the initial research into audiovisual translation focused on technical and linguistic issues (Taylor, 2013), while the image and the sound were secondary topics even though these layers are likely to convey meaning that will influence the audience’s interpretation and reception (Balirano, 2013; Pérez González, 2014, p. 122). There may be different reasons why authors have paid less attention to the multimodal features of audiovisual products (including the industry’s focus on language, Pérez González, 2014, p. 185), although the difficulty to collect a corpus containing information about all the different layers of multimodal texts is certainly one of them. In this respect, Pérez González listed twenty-three sub-modes for the core mode ‘language’ (2014, p. 199), six for the core mode ‘sound’ (p. 206), twelve for the core mode ‘music’ (p. 208) and fifteen for the core mode ‘image’ (p. 214). For his part, Yves Gambier (2023) has recently listed fourteen different audio and visual codes that contribute to shaping films as audiovisual products. The interaction between these different codes and/or modes will determine the generation and interpretation of meaning in the source text and, therefore, needs to be considered in the translation process.

The focus on the linguistic and technical aspects of films and television programs has gradually changed over the past two decades as researchers have acknowledged the significance of the sound and the visual layers. Chaume Varela underscored (2004, p. 12) that the translation of filmic products needs to incorporate the ways in which different signifying codes produce meaning. In the same way as translators of audiovisual material are constricted by the time available, other elements need to be part of the translation process in order to produce meaning. As Pérez González (2014, p. 122) puts it, in this type of translation ‘translated language is meant to act as the mortar that cements the rest of the semiotic blocs together’. Hence the importance of considering all the layers in the translation of the verbal component. Take, for instance, the following brief exchange from the classic movie The Ghost and Mrs Muir (1947):

Mrs Muir: My husband is dead

Miles Fairley: Oh! Oh!

In the scene, Miles Fairley had been flirting with Mrs Muir, whose title indicated that she was married. Mrs Muir’s words caused two fast reactions that are verbalized by means of the same interjection. The intonation used by Fairley, falling in the first case and rising in the second, conveys two distinct meanings. The former, as could be expected, is used to express sympathy at the news, while the latter indicates that Fairley realized that new possibilities opened before him. The translation of the exchange will need to take into account the range of meanings encapsulated in these two very short words uttered in a very short period. In addition to the suprasegmental elements present in the source text, Fairley’s facial changing expressions will also help the translators. In sum, this brief example illustrates the significance of at least three layers (linguistic, oral, and visual), all of which generate meaning in the source text and will need to be considered for the adequate translation of the verbal component.

However, as mentioned, the study of the multimodal nature of audiovisual products poses problems for the collection of the material to be analyzed and its many features. An interesting approach was proposed in 2000 by Paul J. Thibault, whose multimodal transcription encompasses the visual frame, the visual image, the kinetic action, and the soundtrack. Breaking down a text into these components would allow the translators to fully understand the meaning of the source text and, therefore, adapt it for the target audiences. Interestingly, Thibault is primarily a linguist that has worked with Saussurian (Thibault, 1997) and functional linguistics (Thibault, 2021), but he has also investigated different elements of human verbal communication, such as the importance of one’s voice (Thibault, 2018). Thibault’s proposal has been used in audiovisual translation research (Reviers, 2019; Taylor, 2003), although not widely.

In this thematic issue, Siwen Lu’s article proposes a methodology for the study and translation of swearwords clearly related to Thibualt’s work. In ‘More than words: a multimodal analytical framework for studying the subtitling of swearwords’, Lu examines a topic that has attracted much attention in recent years, namely the challenges of translating offensive language, a type of lexis that has been the object of study of pragmaticians. In Translation Studies, the interaction between pragmatics and translation was first analyzed in an excellent collection edited by Leo Hickey (1998), in which contributors considered the importance of cooperation, politeness, presuppositions, and perlocutionary actions, amongst other topics. None of them studied the translation of swearwords. In the past decade, however, the study of swearwords has expanded to Translation Studies as researchers have looked at how offensive language is translated in a variety of settings, often in literary and audiovisual material. In fact, the prestigious Journal of Pragmatics has recently, and almost simultaneously, published two special issues devoted to swearwords, one from the perspective of pragmaticians and another one from the perspective of translation scholars. The latter, which I have guest edited, includes articles by Marie Noelle Guillot, Maria Pavesi and Maicol Formentelli, Annjo K. Greenall and José-Javier Ávila Cabrera among others.

In her contribution to this thematic issue, Siwen Lu studies swearwords in audiovisual texts, but her focus shifts from the linguistic component to the visual elements in order to provide a more comprehensive analysis which can serve to understand the shifts in the target texts and the strategies used by the translators. Lu draws on functional linguistics and theories of multimodality to propose a model aiming at studying subtitled films. The core structure of the model is based on Kress and van Leeuwen’s visual grammar (2006), and, following in the footsteps of Bordwell and Thompson (2010), Pérez González (2014), and Ramos Pinto and Mubaraki (2020), Lu has added the study of the spoken mode (which includes a pragmatic level), the mise-en-scène mode (pertaining to objects, costumes, etc.), and the subtitle mode. The analysis looks at a total of six US movies containing between 39 and 249 swearwords and finds that the translators resorted to omission and de-swearing as the most common strategies. This seems to be the trend in the translation of swearwords into Chinese, as others have suggested (e.g. Chen, 2004). However, Su claims that, although the Chinese versions tend to avoid the use of swearwords, this does not necessarily entail the loss of the pragmatic force of the originals because other visual meaning-making elements may compensate the loss of linguistic elements. One limitation to this otherwise interesting article is the fact that Su does not consider the strength of the swearwords used in the source texts.

While Su approaches audiovisual texts from a multimodal perspective, this thematic issue comprises several papers that highlight the significance of the visual and/or acoustic modes, although not necessarily drawing on theories of multimodality. In ‘Dubbing feminist discourse in Disney films for Arab audiences’, Samia Bazzi analyzes a multimodal and multimedial product, namely the movie Frozen, from a functional linguistics perspective. The study of feminist discourse in translation has produced an important bulk of research in the past decades, although researchers tend to focus on literary texts. In her article, Bazzi claims that Disney movies now combine powerful images together with feminist messages to present more empowered female characters. In Frozen, these images, she claims, are linguistically realized, but the words are substantially changed for Arab audiences by means of strategies such as relexicalization and the use of different collocations in order to transform the characterization of the strong-willed protagonist into a heroine with less agency and confidence, and more dependent on the male characters. In her view, this type of shifts may serve to perpetuate the traditional role of women in the target cultures. Bazzi’s use of functional linguistics links her research with that of other authors interested in the study of multimodal texts because her work highlights the importance of combining the exploration of multimodality with other well-established approaches already used in Translation Studies. The effects of these changes upon the targeted audiences could be the object of future research, which could serve to ascertain the extent to which the visual elements of a movie or television series can compensate the linguistic changes discussed in Bazzi’s article.

This thematic issue also includes articles that examine audio description in the representation of multimodal texts. Audio description, also known as video description, refers to the verbalization of verbal and non-verbal content in products such as films and television programs, but also in theaters and museums. It has become an important area of research, as governments and institutions aim to cater for blind and visually impaired audiences so that they can enjoy spaces that were originally reserved for those without this type of difficulty. Audio description, defined by the American Council for the Blind as ‘either live or recorded information, provided by a trained describer that provides descriptions of visual components of an event to become accessible to those who are blind or of low vision’ has become an independent field of study (Taylor & Perego, 2022a, p. 1), with stakes in a variety of settings (Taylor & Perego, 2022b).

In ‘Audio description of gender’, Brett Oppegaard and Andreas Miguel discuss the problems emerging from the combination of audio description for social-inclusion purposes and the use of gender labels for people who are unable to decipher visual clues that might help them understand the world around them. After examining over twelve national and international guidelines, the authors found that these are outdated and do not address the complications of gender labeling. Subsequently, Oppegaard and Miguel used Grounded Theory techniques to analyze a total of 179 self-descriptions used by the individuals in the experiment they carried out. The purpose of their research was to clarify both the type of labels used by the respondents and the importance of those labels. The results show that the individuals in the study used gender-related labels, although they were unevenly applied. Interestingly, the labels were almost always used from a third-person perspective. Oppegaard and Miguel believe that this type of research is crucial to produce clear guidelines and make suggestions for further investigation. These recommendations could enhance the audio description experience, but the authors realize that some of them may, in fact, complicate the reception of an audiovisual product.

Game localization

Game localization is another major area of research that may benefit from applying a multimodal approach. Game localization refers to ‘the many and varied processes involved in transforming game software developed in one country into a form suitable for sale in target territories’ (O’Hagan & Mangiron, 2013, p. 19), and it needs to take care of the linguistic, cultural and technical components of the products as well as the legal and visual aspects (Bernal Merino, 2015). In addition, translators must often work with unstable texts, that is, they are not the final versions of the products (Mejías-Climent, 2019, p. 100).

In this thematic issue, Mikołaj Deckert and Krzysztof Hejduk delve into game translation from the perspective of the consumers of this type of audiovisual material, focusing on how user experience can be affected by a certain type of mistakes, namely typos. While Deckert and Hejduk do not approach errors from a theory of multimodality, their experimental study considers aspects such as usability and enjoyment as well as audio and visual aesthetics. In the same vein as Oppegaard and Miguel, they work with a group of consumers and the results show that typos impacted user satisfaction, as players were partly distracted by them. Above all, typos highlighted the need to consider a variety of aspects in game production, some of which may have not received enough attention. Deckert and Hejduk stress that writers, filmmakers, and game creators (as well as publishers and distributors) need to be aware of the importance of translation in the creative process as it is a practice that provides cohesiveness to the different components of the final product. As Mejías-Climent (2019, p. 110) points out, the localization of video games comprises a series of technical and functional aspects that require careful consideration in order to preserve the intended user experience.

Multimodal texts in public spaces and museums

Public spaces, which have not received much attention in Translation Studies, provide an abundance of multimodal texts whose nature and function are ostensibly different from the texts discussed so far. The term ‘linguistic landscape’ has been used in connection with this type of environment shared by public institutions and private companies alike. Landry and Bourhis highlight the importance of these spaces and the relevance of analyzing the ‘visibility and salience of languages on public and commercial signs in a given territory or region’ (1997, p. 23). Because visitors rely on the information contained in public notices, the translation of these texts, which may contain images as well, is of paramount importance for the economy of a city or town and for the well-being of tourists. For instance, Xi Chen (2023) has recently conducted a study of 569 public notices in Macao, a city where Chinese, Portuguese, and English are used, even if the number of speakers of these languages varies. Drawing on Kress and van Leeuwen (2006), Chen studied the ideational, interpersonal, and textual functions of the polysemiotic city signs, and found that, verbally, the signs contain a combination of Cantonese expressions and poetic forms, very common in Chinese public spaces. These verbal elements are combined with photographs, cartoons, and drawings that aim at reinforcing the messages. While translations into English and Portuguese attempt to retain the original features, this is not always achieved, particularly as regards the more poetic components, which turn the target texts into direct speech acts.

As regards museums, these public spaces are places where cultural experiences are also polysemiotic (Sturge, 2007), and where exhibits are presented in combination with a series of objects, including written texts, that aim to enhance those experiences. In addition, modern museums provide visitors with audio guides in several languages. In a pioneering study, Sharon Deane-Cox (2014) analyzed the translated guides at the French memorial site of Oradour-sur-Glane and underscored the role of translation as an intercultural carrier of memory (2014, p. 273). Although Deane-Cox did not study the interaction between language and visual elements, she points out that the English translations of the French guides make the synthesis between the auditory and the visual elements less discernible (2014, p. 280). Similarly to Deane-Cox’s work, Spiessens and Decroupet (2023) have considered the interaction between translation and memory, although their work emphasizes the significance of the visual.

For her part, Min-Hsiu Liao (2023) contends that written texts are an integral part of the museum experience and, therefore, it is of great importance to analyze the selection of the texts as well as their relationship with other meaning-making elements. Her study focuses on a permanent exhibition in the Opium War Museum in Dongguan, China, where the objects are combined with text panels, audio guides, subtitling, and wax figures. Through the lens of resemiotization, Liao shows that source text material is de/recontextualized to be integrated in the discourse of the museum, which remains highly institutionalized (2023, p. 60), through a variety of semiotic layers including speeches and photographs. The resulting English texts tend to be critical of the British, but usually quoting foreign language sources rather than Chinese sources, perhaps in an attempt to remain objective or to accentuate the injustice of the war from the perspective of non-Chinese sources.

To allow visually impaired visitors to enjoy exhibits and exhibitions, museums are increasingly providing audio description options (Soler Gallego, 2012). Several publications have proposed methods to improve their quality and/or have analyzed their features. For instance, Elisa Perego (2019) probed a corpus of eighteen audio descriptions and found that the texts complied with the recommendations only partially. They included vivid and informative narrations but tended to be more complex lexically and syntactically than recommended. Although Perego stressed the need to provide appropriate language to replace visually intense elements, the focus of her work was not the interaction between the visual, the acoustic, and the words.

In addition to Oppergaard and Miguel’s article on the importance of audio description of gender in audiovisual products such as films and television, this thematic issue includes an article on the use of audio description in museums. In 2019, Rachel S. Hutchinson and Alison F. Eardley had already explored this topic and emphasized the need to expand the notion of source text in order to analyze this practice in museums. In their view, audio description in this setting should go beyond the visual elements of museums’ collections, and consider the social, cognitive, and emotional elements that may be part of the museum experience as a whole. Looking at it from the perspective of psychology, Hutchinson and Eardley (2019, p. 54) stressed that audio description should pay increased attention to the experience rather than to the objectivity of what is described in order to allow the recipients to construct the meaning and create their own texts. In a different piece, Hutchinson and Eardley (2020) investigated the priorities of audio describers in twelve countries and found important differences concerning what should be incorporated into descriptions.

In line with Hutchinson and Eardley’s research, in ‘Scripted or spontaneous? Two approaches to audio describing visual art in museums, Maija Hirvonen and Betta Saari compare scripted and spontaneous descriptions of exhibits that are delivered in live interaction with users. This interaction involves at least two participants in a multimodal environment where cultural translation takes place via the use of inter-, intra-, and intermodal translation and interpreting. Hirvonen and Tiitula (2019) had previously proposed the use of multimodal conversation analysis to study translations. In the article included this thematic issue, Hirvonen and Saari follow a similar line of work and analyze the recordings of two guided tours in Finnish art museums with live audio description. The recordings were transcribed to describe the various modes present in the tours (speech, gestures, etc), and the material was complemented with interviews with the participants. Their findings show the value of audio description for the visually impaired in both cases, although the spontaneous approach seems to provide more room for interaction. As Hutchinson and Eardley (2019), Hironen and Saari emphasize that the experience is the prime objective of a museum visit, although they do it from a different (but complementary) perspective.

Comics

Comics have received limited attention in Translation Studies (Barodo, 2015; Kaindl, 2004; Zanettin, 2008). Kaindl (2004) focused on the interplay between verbal and visual elements in the translation of humor and underscored the problems that comics posed for translators, as some images can be culturally specific and, therefore, difficult to integrate in the translation of the verbal elements. In fact, he stressed that the belief in the existence of a kind of international visual ‘Esperanto’ is unfounded (2004, p. 183) and that images are governed by the conventions of the culture in which they are produced. For this reason, the interpretation of gestures, visual expressions, and so on can only be correct if the translator is familiar with the significance of these elements in the source and target cultures. Drawing on Kaindl’s work, Barodo (2015) analyzed the translation of the Franco-Belgian comic series Thorgal into Polish and found that visual compensation and typography were used by the translator. Although Kaindl’s and Barodo’s work provides interesting insights into the importance of the image in the translation of comics, they do not draw on multimodal theories, highlighting the fact that the study of multimodal texts may benefit from the intervention of several approaches.

In this thematic issue, in the article entitled ‘Thinking-for-translating in comics’, Rosa Alonso analyses a very linguistic topic regarding the English and Galician versions of the French comic Asterix, namely the types of motion verbs used in the three languages and the strategies used to translate the so-called Manner and Path of motion verbs. Alonso’s study combines qualitative and quantitative approaches and draws on the work of Slobin (1996, 2022), who has been previously used to analyze the translation of novels such as the Harry Potter series. Although Alonso does not draw on theories of multimodality either, she finds that, among the various strategies used by the translators, ‘visual compensation’ is one of them. This underlines the fact that translators are aware of the significance of the images in order to produce adequate translations. In Translation Studies research, linguistic compensation has been widely discussed, while visual compensation has received far less attention (e.g. Barodo, 2015; Molés-Cases, 2020).

This thematic issue includes a second article devoted to the translation of comics. In ‘Translation and the acquisition of symbolic capital’, Nicolas Martinez delves into the translation of the Francophone Blueberry comic series by scriptwriter Jean-Michel Charlier and artist Jean Giraud, successful in countries such as Italy and Spain, but less so in Anglophone areas. Martinez, who focuses on the American editions, combines the findings of interviews with the translators and editors of the series with the study of a selection of textual samples, including the visual elements. Martinez draws on Gouanvic’s work on the sociology of translation and illustrates how the decision to publish certain volumes of the series rather than others has had an impact on the approach to the translation of the volumes. More importantly, Martinez examines the network of agents involved in the production of the English versions. These agents had to make decisions concerning the original artwork, the coloring, and the lettering. Together with the translators and Jean Giraud himself, they helped to shape the final English versions. Interestingly, the translators were fundamental in the publication of these books, as they had specialized in the translation of comics and were involved not only in the linguistic transfer but also in the production, highlighting the symbiosis of images and words in the creative process. In fact, as Martinez points out, the American translators were cartoonists and comic editors and had a creative and monetary interest in achieving the success of the translated versions. The article illustrates, thus, the importance of considering the various modes of multimodal texts together with other aspects such as the agency of those involved in the translations.

Other cultural products: Chinese opera and Chinese reality shows

This issue of Perspectives also features an article devoted to the retranslation and public performance of two classic Chinese operas. While the study of translated plays has attracted considerable attraction, especially from an adaptation studies perspective (e.g. Krebs, 2014; Zatlin, 2005), the importance of the visual elements in the performance, and hence translation, has gradually become an object of interest for translation researchers. For example, Carpi (2020) has studied the semiotic complexity of songs in stage musicals, where lyrics are used by the characters to communicate information as well as emotions. Carpi examines how the verbal, audio, and visual modes interact with each other to produce meaning and propose a method of analysis that includes the study of clusters, repetitions, the cultural background, music, interludes, and pauses as well as dances and embodied behavior. She finds that songs tend to be more open to interpretation than the more conventional parts of the dialogues and, therefore, attention should be paid to other elements to convey all the nuances of the source texts.

The visual and acoustic elements are also of great significance in the case of Chinese plays and especially Chinese opera. Huang and Wang (2021) have studied the terminology used in the translation of Peking Opera terms taking into account the verbal and visual signs used to convey information to the target audience. Drawing on Kress & van Leeuwen’s research, Huang and Wang posit that although the images are adequately distributed, they are not placed in the right areas to complement the verbal elements, which might undermine the purpose of the translation, namely the promotion of a cultural product that has been declared Intangible Cultural Heritage by the UNESCO. In this issue, Ann-Marie Hsiung analyses the retranslations of the classic operas The West Wing and The White Snake as performances for audiences rather than texts for readers. Combining features of multimodal approaches and the so-called ‘Outward Turn’ in Translation Studies (see Bassnett & Johnston, 2019), Hsiung looks at these Chinese operas as examples of one of the most multimodal literary and cultural genres because of the significance of non-verbal modes such as singing, musicality, and stage movements. As specialists in theater and Asian cultures, the retranslators were aware of the relevance of the non-verbal elements for contemporary English-speaking audiences, which allowed them to create plays that were both entertaining and educational by means of innovative scenes and singing styles, although the reception of the more daring adaptation of The White Snake was received with reservations by Chinese audiences and critics. Hsiung concludes that, in any case, it was the additional roles of the translators as directors and scholars that led to the success of the adaptations of these multimodal texts.

Finally, this thematic issue includes an article that delves into the nature of impact captions (not to be confused with ‘closed captions’, the term used in the US for subtitling), that is, a type of intra-linguistic subtitles used for different purposes in audiovisual products in Asian cultures, typically to enhance the audiovisual experience. Impact captions resort to a variety of fonts and colors for this purpose and are normally displayed at the top of the screen, in contrast to the traditional position of commercial subtitles. In fact, its use has been controversial (O’Hagan & Sasamoto, 2016, p. 33), as viewers are not always satisfied with this type of additions. In ‘Impact caption translation on a streaming media platform’, Yean Fun Chow studies the English and Malay subtitles of a Chinese reality show. Drawing on Park (2009), who studied the use of captions in Korea, and on Bezemer and Kress (2016), who, in their analysis of multimodal texts, employed concepts such as semiotic change, transformation, transduction, agency, losses, and gains, Chow identified a total of 218 captions in the episode analyzed here and posits that the translators might have considered the needs of the target viewers when making the changes in the target texts. The study also shows that overlapping occurs, which may confuse the target viewers, underscoring the fact that captions can minimize rather than maximize reception.

All the papers selected for this thematic issue of Perspectives examine multimodal texts, that is, texts combining the verbal component with at least another meaning-making mode. In all cases, the contributors discuss the significance of these modes or stress the importance of taking the non-verbal elements to retain the implications of the original for the target audience. However, as is often the case in Translation Studies, not all of them draw on multimodal theories (Adami, 2023). Even contributions to special issues or collected books devoted to multimodal texts do not necessarily use theories of multimodality to analyze the texts of choice (see, for example, the articles in the special issue edited by Tuominen et al., 2019). This illustrates the fact that the analysis of multimodality and translation can be enriched by a variety of approaches and that is necessary to consider aspects that have not been studied by researchers interested in multimodal texts.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Roberto A. Valdeón

Roberto A. Valdeón is a full professor at the University of Oviedo, Spain, an honorary professor at South China Business College, Guangzhou, China, and a research associate at the University of the Free State, South Africa. He is a member of the Academy of Europe.