Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image

Abstract

Clinical synesthesia is commonly defined as the experience of having perceptions in one sensory modality triggered by a stimulus from another. This paper adopts a particular orientation toward synesthesia, exploring it as a cultural phenomenon common to us all, as an ability that can be learnt instead of an accidental neurological condition. If synesthesia is both a capacity that we are not fully aware of and a way to access what is stored in memory even at the unconscious level, can art help us to bring this awareness back? Bearing upon a close reading of selected artworks created by Johannes Deutsch, a multi-media artist who has been experimenting with synesthesia, the paper argues that synesthesia can become a tool in the hands of contemporary artists to revitalize the Wagnerian ideal of a “total work of art”. This is to be understood as a politics of the senses based on communality rather than individualism, not as an ideology of totalitarian tendencies. Ultimately, the transformative potential of certain art resides in its capacity to foster a pedagogy of the image that is based upon multi-sensoriality, memory and history.

Keywords:

• Synesthesia
• multi-media arts
• neuroscience
• Agamben
• memory
• total artwork

In nova fert animus mutatas dicere formas/corpora

(“I intend to speak of forms changed into new entities”)

Ovid, Metamorphoses, Book I, lines 1–2.

1. Introduction

How would it feel to taste the color “navy blue,” to smell the scent of “redness,” to see the noise of a shrill car horn? Would such an ability influence or change our relationship to the world or to other sentient beings? This type of experience is based upon the simultaneous cross-stimulation of different perceptual sensory systems, an experience at odds with our need for order, systematic organization, division and categorization of our perceptions of phenomena in the world.

Peculiar as it might appear, some type of cross-modal perception is nevertheless a common experience: we can feel goose bumps not only when it is chilly but also in the presence of disturbing or emotional images or when hearing the sudden scratch of chalk across the blackboard. This means that we are all synesthetes, albeit not in a strictly neurological sense: as Ramachandran and Hubbard (2001b) demonstrated through a Gestalt test on non-arbitrary mapping between speech, sounds and the visual shapes of objects, we all have some synesthetic linking, and we all experience one type or another of potentially involuntary synesthetic perception.¹ The persistent belief that the different sensorial inputs coming from the world, such as sounds, smells and images would each be perceived by a different sensory organ, such as the ear, the nose or the eyes, and then transmitted to the area of the brain devoted to the specific sensory modality is no longer sustainable.² Our perception and experience of the world is multi-sensorial. Senses emerge after sensory stimuli extend into and co-mingle in brain areas rather than being immediately available from the so-called sensory organs externally visible on the body (van Campen 2010, 100, Ramachandran and Hubbard 2001b, 979–983). This is the case not only for the way in which we perceive the world, but also for our learning habits. Studies in teaching and learning methods have discussed cross-modality as one way in which our learning experience can be improved (Williams, Gumtau and Mackness 2015).³

This article adopts a particular orientation toward synesthesia, responding to Whitelaw’s call for exploring synesthesia as a cultural phenomenon (Whitelaw 2008) with the potential of enacting a transformation in the way in which we interact with the world. Bearing upon a close reading of several artworks created by the multi-media artist Johannes Deutsch, along with an analysis of synesthesia as an ability that can be learnt gradually rather than just an in-built neurological condition shared by a small number of individuals (van Campen 2010), it becomes clear that the transformative potential of synesthesia cannot be fully understood if analyzed exclusively using the concepts and findings of neuroscience. Interestingly, the ancient Greek meaning of synesthesia is rather different from the modern scientific explanation. Namely, in ancient Greek the prefix “with”, which accompanies the verb “to sense, to perceive” (aisthanesthai), gives the word synesthesia the nuance of perceiving together with others rather than an individual cross-modality type of perception, turning synesthesia into a form of “feeling in common” (Heller-Roazen 2007, 81). Before being circumscribed to a purely neurological understanding, therefore, synesthesia coincided with a shared capacity of perceiving one’s own perceptions, anticipating Merleau-Ponty’s (1969) chiasmatic intertwining of the two hands touching each other.

Merleau-Ponty argued that “synesthetic perception is the rule, and we are unaware of it only because scientific knowledge shifts the centre of gravity of experience” ([1945] 1962, 229), opening the way for an understanding of this phenomenon as a normal, quite common brain function emerging from our shared biological roots. Synesthesia is a preconscious synesthetic ability and experience shared by everyone – an experience similar to a stream of sensorial impressions that language organizes in distinct sensory domains, whose division (touch, smell, sight, etc.) and taxonomy have been superseded (van Campen 2010; Howes 2003). Lost to conscious awareness except for clinical synesthetes subjects, our synesthetic ability is a condition typical of mammalians in response to the adaptive demands of living in this world: “I call synesthetes cognitive fossils because they are fortunate to retain some awareness, however slight, of something that is so fundamental to what it means not only to be human, but mammalian!” (Cytowich 1993, 167).

If synesthesia is both a capacity that we are not always fully aware of and a way to access what is stored in memory even at the unconscious level, can art help us to bring this awareness back? I argue that synesthesia can become a tool in the hands of contemporary artists to revitalize the Wagnerian ideal of a “total work of art”.⁴ This is to be understood as a politics of the senses based on communality rather than individualism, not as an ideology of totalitarian tendencies. A politics of the senses based not only on cross-sensorial perceptions, but also on arbitrary linkage in cross-subjective/cross-sensorial patterns of perception, through which a work of art is integrated with our organism, psyche, culture and environment. Artwork which aligns these patterns is potentially transformative. Synesthesia is one of the means in which a different politics of the senses could be created, allowing an individual to perceive cross-sensory Gestalt patterns, become more aware of the hidden correspondences among ourselves and the physical environment, and thus interact with it differently.

The in-built shared synesthetic ability of common-sensing and “feeling in common” complicates, at least at the theoretical level, the often a critical celebration of interactivity. Specifically, interactivity is often understood as being based upon technological mediation supposedly capable of enhancing not only our interaction with each other, but also with the object, such as a work of art. Furthermore, synesthesia may open up possibilities of coping with our existential Western condition of acceleration and excess, which is commonly understood as a sensorial and cognitive over-stimulation alongside constant information flow, all of which are features of today’s political, cultural and artistic globalised world. Our response to these sensorial and cognitive inputs, solicitations and stimuli is at the centre of the theoretical and artistic preoccupations of Deutsch. My objective in discussing examples of Deutsch’s art production is not to offer a critical reading of his oeuvre, but to examine his art as a critical function helping us to reframe synesthesia in broader cultural and historical terms, rather than just as a purely neurological condition. I argue that his engagement with the phenomenon of synesthesia is an attempt to organize and govern excess rather than being overwhelmed by it; his work transforms excess into an opportunity that frees us from the disorientating burden of constant stimuli, such as those flowing from a high tech audio visual culture. Through synesthesia, Deutsch creates a pedagogy of the image that is based upon multi-sensoriality, memory, and history.⁵

2. Synesthesia: moving beyond sight

Both the arts and philosophy engaged with the phenomenon of synesthesia well before any scientific approach to it was developed. Although a detailed analysis of the various configurations of synesthesia falls outside the scope of this article, it is worthwhile pointing out key moments in the history of this phenomenon. In his archaeology of sensation, Heller-Roazen examines several instances across philosophy and the human sciences in which the concept of sentient beings came to the foreground. For example, he recalls how the verb sun (with) and aisthanesthai (to sense or to perceive) is the term applied by Aristotle to “a perception shared by more than one” (Heller-Roazen 2007, 81), a common faculty by which animals and human beings sense that they are sensing. This awareness to exist and to perceive, which is not yet self-consciousness, is “the natural accompaniment to the execution of every act of sensation” (84).⁶ The etymology of the word synesthesia makes clear that it is a concrete type of perceptual experience rather than a form of mental imagery. In the nineteenth and first half of the twentieth century’s, arts, poetry and literature have popularized synesthesia. Philosophical thought is often skeptical if not dismissive of senses other than sight, with few notable exceptions, including Merleau-Ponty, Jacques Derrida, Michel Serres and Jean-Luc Nancy who devoted their critical attention to touch and hearing and to the intertwining between the senses.⁷

The return to the senses in contemporary philosophy has been mirrored by the so called sensory turn in various disciplines in the humanities and social sciences, from anthropology, through cultural and museum studies, history, geography, film and visual culture.⁸ The scientific community has long since stopped considering synesthesia an illusion, granting it the status of a neurological phenomenon. Although there is no shared scientific theory on the origin of synesthesia, nor a definition capable of grouping together different phenomena that can be labeled as synesthetic, scientists agree that synesthesia is a neurological condition that is believed to have a genetic basis in which the stimulation of one sensory pathway triggers an involuntary, automatic response in a second sensory pathway (Asher et al. 2009). Synesthesia can result both from a perceptual activity – for example, seeing – and from a cognitive activity – for example, reading or listening to music. A triggering stimulus, called an inducer, must occur for synesthesia to take place, whereas the second sensory pathway triggered by the inducer is called concurrent (Grossenbacher and Lovelace 2001).

Currently, the definition of synesthesia varies depending on the framework adopted to investigate it. In the 1990s, the English neuropsychologist Simon Baron-Cohen was one of the first to systematize the scientific work conducted on synesthesia, agreeing with other scientists upon the following definition:

We (…) define synesthesia as occurring when stimulation of one sensory modality automatically triggers a perception in a second modality, in the absence of any direct stimulation to this second modality. (…) Many combinations of synesthesia are reported to occur naturally (…) Our experience suggests that coloured-hearing synesthesia is by far the most common form and that certain combinations of synesthesia almost never occur (for example, touch to hearing) (1997, 3).

During the 1980s, Baron-Cohen sought to design an objective test for detecting genuine synesthesia at a time when no diagnostic test was available. Before him, Cytowich outlined five features that would allow the diagnosis of synesthesia: first, although involuntary, synesthesia must be elicited; second, synesthesia is durable – it endures a lifetime; third, it is projected onto the outside world rather than being confined in the mind; fourth, it is memorable and can be triggered back by the presence of the same type of stimulus and, lastly, it is ecstatic, close to a mystical experience (Cytowich 1993, 76–79). Van Campen discusses refined and improved versions of the Baron-Cohen’s test, pointing out, however, that it is very difficult to design a scientific method not only to distinguish between the two categories but even between different types of synesthesia (2010, 124). Marks and Mulvenna (2013) have distinguished six forms of synesthesia: cross-modal correspondence, cross-modal imagery, cross-modal sensory, autobiographical memory, empathic perception, hallucination, and the Doppler illusion.

Brain imaging techniques made it possible to investigate synesthesia as a neurological phenomenon located in the brain. The regions of the brain activated by synesthetic perceptions are the same ones that get activated by ordinary perceptions. According to Ramachandran and Hubbard (2001a), synesthesia is a perceptual mode based on cross-wiring between different senses and different areas of the brain devoted to the senses. Being involuntary and fixed, synesthesia is genetically rooted and results from the evolution of particular areas of the brain. Ramachandran, however, argues for the existence of a synesthetic ability which operates beyond synesthetic perception and identifies patterns across the senses that create modality-free abstractions such as language, creativity, metaphorical reasoning, mathematical reasoning, etc.⁹ In this respect, synesthesia can also be induced and learned, as not only scientists but also artists have understood.¹⁰ It is possible to design synesthetic experiences across different sensory modalities in highly artificial, technologically-mediated environments capable of stimulating in the observer/user a cross-modal, embodied behavior and learning.¹¹

Synesthesia is a medium at the disposal not only of artists and designers but even of companies interested in eliciting a certain type of response in potential clients. For example, knowing how to induce and control synesthetic experiences is of crucial importance for marketing products in the automotive industry. During the course of the interview I held with Deutsch in Wien in September 2012, he touched upon this topic by mentioning that one of the collectors of his artworks is the patron of Ford who revealed to him the huge interest of car companies in research and design based on synesthesia. Especially in luxury cars, each element is carefully designed to trigger synesthetic responses, from the materials used for the steering wheel and the dashboard, the closing sound made by the car door, to the fabric of the carpets, in such a way that you decide to buy a car from what you feel in your stomach.¹² Far from being a scientific phenomenon strictly analyzed within the laboratory, induced synesthesia can be put at the service of capital, as a way of driving our bodies and brains into certain comfortable perceptual experiences, associations that, ultimately, encourage us to desire and consume. It is crucial, therefore, to be aware of the power implied by the ability to induce and control synesthetic experiences.

The universe of synesthetic perception is, however, not necessarily mediated by technologies as examples of modern art and literary exploration of synesthesia demonstrate. The following examples show how synesthesia gained momentum as a means to create a total work of art capable of appealing to the audience across sensory modalities. The relationship between sounds and colors, for example, was explored by Pythagoras in the antiquity, by Leonardo da Vinci in the Renaissance and by writers and artists working in different media in modernity. A particular case is that of the Russian artist Wassily Kandinski, a member of the German artist group called Der Blaue Reiter (the Blue Rider) and a teacher at the Bauhaus at the start of the twentieth century.¹³ Kandisky’s own 1912 opera, Der Gelbe Klang (The Yellow Sound) was a mixture of color, light, sound and dance, aimed at reaching out for the spectator to generate an experience closer to a state of trance (Doran 2013). The fusion of the senses was Kandinski’s pathway to transcend representation and turn his canvases into compositions, improvisations closer to music and sound rather than to painting and vision. Regardless whether Kandiski was a true synesthete or not (van Campen 2010, 56), Kandinski’s research and art practice on the correspondence between colors and music were inscribed within the first synesthetic experiments conducted by the Der Blaue Reiter group with the goal of creating a total work of art capable of transcending any medium-specificity.¹⁴ Painters, writers, musicians such as Klee, Nabokov, Rimbaud, Baudelaire, Sibelius to name just a few, have also described various types of non-technological mediated synesthetic experiences that unleashed the creative forces of the unconscious, albeit not necessarily using the term synesthesia.¹⁵

Synesthesia thus becomes a mode to overcome the limitations of a specific artistic language and medium in order to absorb others, moving toward a total work of art capable of cross-stimulating our senses. Synesthesia should therefore not be regarded as a merely perceptual phenomenon describing a neurological condition, a deviation from “normal” sensory perception. In synesthetic artworks, induced synesthesia is an active sensory modality that requires the direct intervening of the onlooker who tunes in to the artwork. Synesthesia allows us to develop a cross-modal, sensory-based theory of interactivity that is not simply triggered in the form of a visual or tactile stimulus-response mechanism in the audience, but at cognitive and cross-modal multi-sensorial level. If it were possible to embrace the ancient understanding of synesthesia as a form of common, shared perception that can be learned together with the modern scientific concept of synesthesia, one could argue for the sensorial biological roots of a community of the senses that might help us to face the challenges posed by the global culture of excess, increasingly characterized not by the physical circulation of commodities, but by the circulation of information and raw data (Rodowick 2001). Learned synesthesia might be a critical function to think through possible ways in which excess can be channeled and controlled.

3. Synesthesia and synthesis: toward the total work of art

In this section, the exploration of synesthesia is conducted through the lens of the “total work of art”. The reunion of art, design and architecture in a search for synthesis was the driving force behind the experimental projects conducted under the auspices of the Bauhaus, a laboratory that aimed at changing society and shaping a modern type of human being and environment.¹⁶ Offering a counter-thesis to the reading of modernism as characterized by the division and medium-specificity of the arts, Roberts (2011) argues how the concept of the total work of art is pivotal to the development of several avant-garde movements of the twentieth century. He sets out to demonstrate the thesis of the political value played by synthesis, which is central to the concept of the total work of art and, consequently, to the aesthetic and political radicalism of avant-garde movements. Roberts wants to demonstrate the relevance of the concept of the total work of art not only for the arts in modernity, but also for modern politics.

The idea is that artistic synthesis, the aspiration to, or the “tendency” toward, the total work of art, was never purely an aesthetic concern but a desire for social, cultural and political renewal (Roberts 2011, 1). Der Hang zum Gesamtkunstwerk (The Tendency toward a Total Work of Art), the encyclopedic exhibition curated by Szeemann in 1983 at the Kunsthaus in Zürich still works as a reference model for contemporary artists and curators, including Johannes Deutsch.¹⁷ Before going into details of how Deutsch creates synesthetic effects in his artworks, his aesthetic and creative concerns will be briefly discussed. If Deutsch is first of all a painter as he defines himself, his art production refuses any strict categorizations in terms of genres or media specificity. In his own words:

I did not know that maybe later I would be involved in new media so I describe now computer as a tool, as a strategy, not as a new medium; I describe computer as an interaction strategy, as a credit to mass society that prompted us to do art in a different way. It all began because of the Bauhaus, because already as a student I felt how precise my tools were in relation to my imagination. Let us make the famous example of the apple. If I want to design an apple I must eat this apple, I must describe this apple in my language, I must really study the apple by drawing it, building an apple from clay, and then maybe photograph my apple, and then painting the apple and so on.¹⁸

This passage helps understanding how the Bauhaus constitutes for Deutsch a departure point to explore new media in dialogue with the cultural, artistic tradition and with cutting-edge technologies. The methodological example of the apple, which is perhaps reminiscent of the lively presence of Cézanne’s Still Life with Apples (1895–98), demonstrates how Deutsch explores each different medium according to its peculiar features and possibilities without, however, randomly mixing up genres and techniques. Every brush stroke, every detail, every animation merges with the whole preserving, at the same time, its specificity. The synthesis is achieved by Deutsch only after roaming from one medium to another which is documented by his sketch books, drawings and storyboards. The suspicion of medium purity does not mean that Deutsch neglects the peculiarities of each medium, quite the opposite: each medium is explored in all its rules and possibilities, then pushed to the extreme.

The starting point for Deutsch is neither virtual reality, nor the digital environment, but the complexity and uniqueness of his own life situations, the blurred, tangled mass of experiences and familiar faces.¹⁹ Refusing the notion of medium purity, Deutsch creates spaces that are experiential, imaginative and imbued with memory layers. Contrary to any understanding of computer-based art as being almost exclusively tied up to the present time, to the “here and now”, Deutsch shows the intricate intertwining between layers of time, past and present histories and perceptual modes, thus reframing the complex relationship between art and science in light of history and culture. In what follows, I discuss three pieces of Deutsch’s art bearing upon the critical framework of synesthesia and of synthesis of media, styles, and cultural references. The artworks are three variations on a single theme, that of synesthesia.

Toward the end of the 1980s, Deutsch was experimenting with new technologies in order to insert movement into the still image of a painting, transforming the spatio-temporal dimension of the image. Gesichtsraum (Face Space), is the first interactive installation created by Deutsch with CAVE, a virtual reality application. Computer-based paintings are moved into a real space, a cube in which projected images change in relation to the movements of the audience. Gesichtsraum, a walk-in space, was produced by the Future Lab in Linz and exhibited in 2002 at the Ars Electronica Festival. It consists of a physical closed cubic space whose borders are created by images projected onto a wall-like transparent film. Spectators can decide whether to experience Gesichtsraum passively, as if they were standing in a 3D cinema or more actively, interacting with the space through bodies. Knowing that sensory perception responds particularly well in dynamic situations, the movements and gestures of the spectator walking through Gesichtsraum can modify the agglomerate of colors and space. Colors become material sounds and shapes: red, yellow, white shapes leave the cube face to migrate to the other parts of the cube. The spectator-director interacts with those shapes, either slowly moving across the space or with sudden short movements, and the traces of her movements become visible in the pictures. At times, all faces of the cube are filled with colored shapes that react to determinate impulses, such as movements and acoustic signals. One starts to hear with eyes and see with ears as much as an orchestra conductor does. No form represents anything in particular, they are forms of feelings, of a certain stimmung (mood) (Figure 1).

Figure 1. Johannes Deutsch, Gesichtsraum, Scene 1, 2002. Image rights: Ars Electronica Futurelab Courtesy of the artist and Ars Electronica Futurelab.

The immersive experience is possible thanks to the computer which allows the artist to reenact the process of transforming a mental image using a pre-existing visual repertoire, thus making the process of creating a digital picture a type of mental activity. The correlation between the interior space of the psyche and the exterior space of the cube is made possible by the movements of the visitor who wears stereoscopic glasses to see everything in 3D. A software sensory system tracks the movements of the viewer who is able to create rooms within the main Gesichtsraum. Each movement can be repeated within the CAVE observing the changes in the projected image, in such a way that the movement and the change it triggers are observed first in the mind of the spectator before making the next move. All possible types of interactions are, however, orchestrated by a pre-existing script that determines the order of appearance of the rooms, the elements, the objects that appear and the proportions among them in response to the movements of the viewer. Immersion is, therefore, more existential than psychological because it is not triggered by any identification with a character, rather it is achieved by overcoming the limits of linear perspective to create a layered, multi-sensorial space that corresponds to our contemporary experience of time and space. Space is unstable, illusionary, blurred, it fades away, refusing to be fully mastered by vision.

The exploration of synesthesia continues in several of Deutsch’s artistic productions after Gesichtsraum, not only in his complex opera-based works like Vision Mahler (2006), a multimedia visualization of Gustav Mahler’s second symphony, but also in relatively smaller-scale projects, such as the Unsichtbare Garten (The Invisible Garden). Vision Mahler blends cutting-edge technology with cultural tradition.²⁰ Each note of the symphony is translated into an image, each musical phrase corresponds to a virtual object floating in the space of the theatre. Those objects are 3D sculptures with life and characteristics of their own, specifically designed by Deutsch with the help of computer developers. Music becomes embodied in big colored objects which Deutsch calls “models” that interact with each other, at times dissolving, losing their contours to become light.

It is in the relatively small-scale project, Unsichtbare Garten, however, that Deutsch arranges an experiment in cross-modal perception through movement. At Admond Abbey in Switzerland there is a unique art collection, to which Deutsch contributed, for the blind and for sighted people who explore the artworks through senses other than sight. The 2012 retrospective Beyond Seeing featured a selection from that collection. Each work was meant to be explored both by blind and by sighted people wearing blindfolds. Johannes Deutsch contributed to this exhibition with his Unsichtbare Garten created in the abbey grounds and meant to be “a space in which blind persons ‘see’ more than ‘sighted persons’” (Braunsteiner and Schwab 2012, 11). The installation, consisting of a natural pathway with different regional trees and surfaces, was a tactile, olfactory and auditory feast. The virtual natural environment was carefully circumscribed and detached from the rest of the space by means of a hedge of pines. The goal was to create a shared space for blind and sighted people to co-experience shared perceptions in a new type of virtual and real environment rather than being a garden for the blind. Each sighted visitor had to share the same sensory deprivation of the blind visitor by wearing blindfolds during the garden walk. The organs of touch, smell and hearing were the prosthetic means through which the visitor explored this virtual natural world.

It is in Zeit Perlen (2005–2012), however, that the critical and pedagogical function of synesthesia becomes fully visible. Zeit Perlen (Time Pearls), also realized as a multimedia interactive work for the museum Haus der Musik museum in Wien, is an interactive opera inspired by Ovid’s Metamorphoses.²¹ Metamorphosis is the cornerstone of this cycle where dreams and reality coexist. Metamorphoses are organized around cross-references among the different myths, so Zeit Perlen is organized like a layered ensemble of painting, photography, music and digitally re-worked images, using the principle of continuous transformation. The presence of a female character represents continuity between the different scenarios (Figure 2).

Figure 2. Johannes Deutsch, Zeit Perlen, Video still, 2012. Image rights: Johannes Deutsch Courtesy of the artist.

The small-scale opera has six chapters full of references to European cultural heritage: the first chapter is the ‘Overture’; the second one, ‘Don Giovanni’, is inspired by Mozart’s opera; the third is the ‘Flying Dutchman’ inspired by Wagner’s homonym opera; the fourth ‘Resonance’, then ‘Nostalgia’ based on the ancient Roman myth of Halcyon; and then the conclusion, the ‘Finale’. Deutsch designed the dramaturgy, the scenes and all the visuals, directing the video and developing the concept for interaction; the software development and the interaction design were made by Contraire (Linz) and the sound installation by Trust your Ears (Düsseldorf). Several musicians from the Wien Philharmonic Orchestra were involved, with the soprano singer Natalia Ushakova in the role of Céleste.

Zeit Perlen is a virtual opera play taking place in a small theatre, with the interacting person standing as the conductor in front of the screen and the audience behind (up to 15 persons at a time). No device is used by the “conductor,” only a natural user-interface analyzing movements via a 3D camera which records all body motions of the interacting person: the conducting arms, the general dynamic of the body, its coming closer or getting further away from the screen, the left or right position, etc. The data collected are analyzed by software and then transformed in an interaction between the dramaturgy, the visual and the sounds. Human gestures define both the variations in the storyline and the complexity of the audio-visuals. As the conductor approaches the screen, the experience is that of walking in the various layers of the scene, where images get bigger and music sounds closer. When conducting via arm movements, more instruments start playing, as well as extra lights and shadows. In this way, for example, the Don Giovanni scene can evoke extra events on to that matter. Like in Deutsch’s first interactive theatre Gesichtsraum, here too the user-interpreter can concentrate on certain things and thus bring them to life. Composing a net of musical objects and visual layers, the user-interpreter turns Zeit Perlen into a quasi living organism that can accelerate or quieten down.

Céleste, who is the central character in Zeit Perlen, sings vocals without any text, purposefully so. The opera is designed for museum visitors who speak different languages and have different cultural backgrounds. The understanding of the play is therefore brought to the audience by means of cross-sensorial analogies, auditory or visual symbols that bear meanings, and iconic couplings. For example, the sound of water refers to images of waves; the sound of fire crackling is coupled with open flames in a cave. These couplings are retrieved from human memory, they are associations that we have established. In the case of symbolic couplings, cultural and semantic correlations are established (Deutsch 2012, 227). These induced cross-sensorial responses in the audience enable a type of engagement with the world that goes beyond classical semiotics.²² (…) at the very basis of the semiotic processes that enable us to make sense of the world there is, for Peirce, perception with its bodily based inferential processes. (…) If the role of the body forms the basis of Peirce’s notion of semiosis, then the same cannot be said for classical structural semiotics, rooted in the work of Saussure and Hjelmslev, where a formalistic approach to meaning was dominant (Violi 2008, pp. 244–245) Deutsch’s work highlights the active role of cross-sensory perception in the semiosis between the artwork and the user-interpreter.

Crucial parts of the brain that are involved in cross-mapping a range of sensory inputs not only give us our unique ability to perceive patterns (over and above associations) across the senses, but also allow us to use this ability and these basic synesthetic perceptual patterns to establish modality-free abstractions (e.g. metaphor, language, mathematics and abstract and creative thought), or patterns of patterns (Williams 2015, p. 49).

The garden and the veranda of Céleste play a major role in Zeit Perlen.

If the user-interpreter stares at the garden then she can go through it, walk on the red carpet and see the little stones in the water: the software system recognizes this act of looking which makes the stones shimmer, glint, shine and the music too glints, the water has a certain movement, it still does not have a naturalistic sound. Deutsch notes:

The little stone is answering my interest, and the music gives me a certain form not an illustration but it gives me something and I recognize it is me, I have a moment of interest in the part of the opera and at the same time the storytelling is going slowly forward, and I fall into something like in a book with a very nice detail and I suddenly re-read it.²³

There is, therefore, a synesthetic play between the stone and the picture and the sound, the stone answers the interest of the onlooker, it serves her gestures and it simultaneously serves the plot. Deutsch calls the veranda his “virtual cube” because in Zeit Perlen the veranda coincides with memory.²⁴ Céleste’s veranda is transformed into a scene where the ice-like sculptures are simultaneously reminders of Dante’s inferno, with a substitution of ice instead of fire, and of the sculptures in the cemetery of Rapallo which appears in the film The Barefoot Contessa (1954). These and other couplings might occur, or not, in the mind of the users-interpreters who not only compose the artwork but are eventually transformed by it. This transformation is interior rather than exterior as is the case of Ovid’s metamorphoses whose main feature is the awareness of the persistence of the first state of being before the transformation in an animal, arboreal or inanimate being occurred. The internal metamorphosis contrasts with the world of excess in which we live (Figure 3).

Figure 3. Johannes Deutsch, Zeit Perlen, Haus der Musik, 2012. Image rights: Iby-Jolande Varga Courtesy of the artist.

Zeit Perlen shows us how induced synesthesia is one possible route toward achieving a greater awareness of our Western shared cultural roots generating an awareness that political, cultural and social transformations cannot disregard an active engagement with history and with our personal histories.²⁵ This awareness is crucial in art that wishes to be truly contemporary and not simply “new”, if one listens to Agamben’s reflections on the relationship one entertains with one’s own contemporary age:

Contemporariness is, then, a singular relationship with one’s own time, which adheres to it and, at the same time, keeps a distance from it. More precisely, it is that relationship with time that adheres to it through a disjunction and an anachronism. Those who coincide too well with the epoch, those who are perfectly tied to it in every respect, are not contemporaries, precisely because they do not manage to see it; they are not able to firmly hold their gaze on it (Agamben 2010, 11).

For Agamben, to be contemporary is not to be confused with being new. To be contemporary means to be able to take distance from one’s own time, to step out of the here and now, to be out of synch with the present moment. Thanks to that, one might be able to see what is at stake in one’s own time. Such an existential condition challenges the linear progressive stream of time. An intense focus on the present, a constant acceleration and excess of stimuli provoke anxiety and alienation. The excess of signifiers, including stimuli, objects, words, events, and sensations confronting us on a daily basis needs be translated into a language that allows us not only to make sense of the world, but also to navigate through it and share the burden of excess with other perceiving subjects. This language does not need to be based on words. It can be based on perceptions, sensations, and synesthesia. Rather than being a metaphor to describe a perceptual process, the synesthetic, morphed layers of psychic landscapes, interior worlds and emotional memories that live beneath the surface of the canvas/screen in Deutsch’s work, explore synesthesia as our forgotten origin, our “hidden sense” (van Campen 2010).

The theme of getting closer to the origin is central to Agamben’s understanding of our relationship with contemporaneity:

Only he who perceives the indices and signatures of the archaic in the most modern and recent, can be contemporary. “Archaic” means close to the arkhē, that is to say, the origin. But the origin is not only situated in a chronological past: it is contemporary with historical becoming and does not cease to operate within it, just as the embryo continues to be active in the tissues of the mature organism, and the child in the psychic life of the adult. Both this distancing and nearness, which defines contemporariness, have their foundation in this proximity to the origin (Agamben 2010, 50).

The past origin that still lives in the present is what becomes visible in Deutsch’s temporally layered art in which to be contemporary means to be able to access history and memory. Synesthesia in Deutsch is what gives us access to our shared origin of ‘cognitive fossils’, of synesthetes. Besides the metaphor of the embryo, Agamben recurs to an astronomical metaphor to explain the concept of contemporaneity: in the universe that expands relentlessly, the most distant galaxies are moving away from our galaxy at a rate such that their light cannot reach us. The darkness of the sky is the light of galaxies moving away from us, a light that cannot reach us because of the high speed of the galaxies from which it comes, a speed higher than that of light. The present time thus happens in the contradictory form of being too early which is simultaneously too late, a mismatch of chronisms. Courage and determination in holding the gaze into the darkness of the here and now are the essential condition of “being contemporary”. It is waiting to be able to perceive a light that, despite being directed toward us, in reality moves away from us and will never reach us. Being blind to the here and now is what makes us attune to contemporaneity.

4. Conclusions

This article has explored synesthesia as a condition that we all have in common and that can be learned. For example, Deutsch encourages us to think about synestetic-based art as a critical intervention to govern excess rather than being controlled by it. Counterintuitively, seemingly random, multi-sensorial stimuli and perceptual overload can be governed through intentionally induced synesthesia. In the guise of an opera conductor, Deutsch orchestrates the multi-media materials he works with in a harmonic ensemble where each medium’s codes are explored, reactivated and reinvented in their specific possibilities. This culminates in the creation of an immersive and transformative experience, coming out from the interaction between the audience and the work of art. The final result is a shared multi-sensory consonance, a refined tuning-in that orchestrates an interactive dialogue between the work’s composition and the audience.

The dream of a total work of art achieved by a synesthetic cross-mapping of visual, auditory, mental and kinestetic modes is, however, never fully achieved, as Zeit Perlen demonstrates: the user-interpreter’s mastery of the complex interplay between music, gestures and visualization into a totality lasts no more than an instant. At first sight, the dream of creating a total work of art might be shared with art-science projects too with their effort to transgress disciplinary boundaries at the level of the research object(s) and in terms of the relationship between research object and subjects.²⁶ However, Deutsch warns us that things might be more complicated than they first appear. Our condition of being “cognitive fossils” (synaesthetes), becomes the neglected root at the base of contemporary art-science projects. Induced synesthesia is much more than just a means to develop interactive works or to pursue a total work of art. Synesthesia is a means to retrieve personal histories and a shared history of ideas. Ultimately, the role of history becomes central because it “generates images and visions of the world” (Braunsteiner 2012, 56), offering us the key to understanding our present time and the images we create.²⁷

Cultural synesthesia is hard to achieve; it remains an unexplored possibility to access what is stored in memory and the unconscious. The synesthetic perceptual play created by Deutsch prompts us to engage with the multi-sensorial character of the image which is also its emotional substrate, its élan vital grounded in the temporality inhabiting each image, as Aby Warburg well understood. The artist creates artwork through cutting-edge technologies and history, adding to the images layers of mythologies and history, both his personal histories and the wider cultural history of various art forms, transforming his images into a living “pathos formulae”.²⁸ The attention driven toward the historical substratus of the image, the attempt to build an archaeology of sensory knowledge is the burden and the responsibility upon contemporary artists, image makers and scholars interested in images and their visual cultures.

Deutsch’s art witnesses an acute awareness of the danger of being left behind in a world increasingly complex and global, a world that eschews our grasp and in which different forms of life – both human and non human, analogue and digital – exist and intermingle. In this respect, being able to sense together is the hidden root of any transformative interaction between us and the world. Furthermore, even in a digital regime characterized by the endless flow of data and information, the synesthetic body becomes an active creator and framer of the multi-sensorial image. This is not a celebration of the Wagnerian utopia of the total work of art, nor an attempt to master synesthesia for the purpose of controlling people’s deepest drives; rather, it is the attempt to reactivate Ovid’s metamorphoses through the lens of synesthesia. Ovid’s metamorphoses underlie a conception of nature in constant transformation, a living archive which makes it possible to perceive the presence of a mythical creature in a tree, in a stone, in a water pond. Similarly, the new mythology that Deutsch sketches positions induced synesthesia as a strategy of resistance to excess, as a radical awakening, a gradual becoming aware of being capable of having synesthetic perceptions and of holding our gaze, firmly, onto the darkness of our time.

Notes on contributor

Silvia Casini is Lecturer in Visual Culture and Film at the University of Aberdeen. Her main research interests concern the aesthetic, epistemological and societal implications of scientific visualization, particularly in the case of emerging technologies. Her articles have appeared in Configurations, Leonardo Journal, Contemporary Aesthetics, Museums ETC, Bloomsbury, Museologia Scientifica, The Italian Journal of Science and Technology Studies. Her first monograph in Italian “Il ritratto-scansione. Immaginare il cervello tra Neuroscienza e Arte” (“The Scan-portrait. Imaging the Brain in-between Neuroscience and Art”) has been recently published by Mimesis (2016).

Notes

1. The original test, called “bouba/kiki” effect, was invented by Köhler (1947). Cytowich too (1993), albeit without providing any experimental testing, shared this belief as illustrated in his book The Man Who Tasted Shapes. Ramachandran and Hubbard’s reflections on synesthesia as a multisensory experience have been further explored by Eagleman and Cytowich (2009).

2. The notion that there are five separate senses is not universal. In the West the number was derived from Aristotle, in other parts of the world the experience is organized differently. For an overview of the cultural history and anthropology of the senses see Howes (1991, 2003) and Classen (1993).

3. The Montessori method is based on shared common synesthetic abilities in children.

4. The term Gesamtkunstwerk (Total Work of Art) was coined by the opera composer Richard Wagner in the wake of the 1848 revolutions occurring across Europe. He first used the word in two essays written in 1849, named “Art and Revolution” and “The Artwork of the Future”, where he called for a reunion of all the arts through music and theatre.

5. Johannes Deutsch was born in 1960 in Linz. He studied graphic art and media art from 1975 to 1980 at the Höheren Lehranstalt für Kunst und Design in Linz and at the Postgraduate Institute for New Media, Städelschule, in Frankfurt from 1990 to 1992. As a painter, director and media artist, Deutsch has always combined media to form a whole. He was already committed to the subject of the total artwork during his time as curator at the Sigmund Freud Museum in Vienna from 1984 to 1989, which finally led him to interactive art. For the 2002 Ars Electronica Festival in Linz, Deutsch created an interactive CAVE installation titled Gesichtsraum, and together with Ars Electronica Futurelab, he staged Wagner’s opera Rheingold as an interactive virtual reality show for the Bruckner Festival in 2004. He continued in 2006 with Mahler’s Resurrection Symphony No. 2 for West German Broadcasting in Cologne. To commemorate Schumann’s anniversary in 2010, Deutsch staged a live cinema production of Manfred (Byron/Schumann) in Düsseldorf including a televised version for ZDF, Second German Television, in 2011. From 2007 to 2012, Johannes Deutsch developed an interactive installation called Zeit Perlen for the House of Music in Vienna. As a visual artist, Johannes Deutsch has presented his works in solo exhibitions, amongst others, at the Vienna Museum of Modern Art in 1992, at Bonn Museum of Modern Art in 1998, at Museo d’Arte Moderna Ugo Carà, in Muggia/Trieste in 2012, as well as in thematic exhibitions at Museum Ludwig, Cologne, in 2000, Salzburg Museum der Moderne in 2006, at the Vienna Leopold Museum in 2010, and in 2012 at the Admont Museum of Contemporary Art. His works are found in leading private and public collections.

6. For his argument Heller-Roazen quotes Alexander of Aphrodisias’ reading of Aristotle’s De Anima (Heller-Roazen 2009, 82–83).

7. Whilst providing an exhaustive overview of the expanding field of sensory studies falls beyond the scope of this article, a few works are among the key references for further exploration. For a critical examination of the multiple scopic regimes at play in French philosophy and literature in modernity and postmodernity see Martin Jay (1993). In his Phenomenology of Perception ([1945] 1962) and The Visible and the Invisible (1969) Merleau-Ponty examines the intertwining of vision and touch in relation to embodied perception. Jean-Luc Nancy devoted a number of studies to both touch and hearing (Nancy 2008, 2007). Derrida (2005) devoted a book on the exploration of touching in relation to Nancy’s philosophical thought. In his book The Five Senses (2008) Michel Serres embarks upon a poetic study on sensory experience as mixed and mingled, although still based on the division among the five senses.

8. The academic interest in the sensory turn is witnessed by the presence of various scholarship and further by the creation of the academic journal The Senses and Society. Among the available collected editions it is worthwhile recalling Howes (2014) in anthropology. Within cultural history, Classen locates the roots of our contemporary Western attempt to recover the senses in the pre-modern cultural interplay between sense modalities other than sight. In her The Color of Angels: Cosmology, Gender, and the Aesthetic Imagination Classen argues how the “visualist regime of modernity” (1998, 1) was then challenged by the attempt undertaken by late nineteenth and early twentieth-century artists to create alternative sensory aesthetics. This was the case, for example, of artists who were part of art movements such as the Symbolism, Futurism and Surrealism. In film and visual culture, the work by Mitchell on images (2002) and by Marks (2000) on haptic images and the senses is a good starting point. To be mentioned is the exhibition States of Mind. Tracing the Edges of Consciousness (Wellcome Trust, London, 4 February - 16 October 2016) which presents a couple of exhibits on the phenomenon of synesthesia both from a scientific and an artistic perspective.

9. The associations between certain sensory experiences are formed very early in life and do not change over time, which means that the condition of synesthesia is quite stable.

10. Against the view of synesthesia as an exclusively congenital condition Daniel Bor et al. (2014) explores the possibility of training adults to acquire synesthetic experiences.

11. For a discussion on the impact of synestetic ability on learning see Williams et al. (2015, 48–54).

12. In his book chapter ‘Synesthesia and Synergy in Art’, Johannes Deutsch quotes many passages from books and articles written by Michael Haverkamp, a Ford Motor engineer whose condition of congenital synesthesia is crucial in his job. See Haverkamp (2009).

13. The Bauhaus was the Weimar school of art and applied design founded in 1919 by architect Walter Gropius.

14. On the notion of medium specificity see Greenberg’s essay Towards a Newer Laocoon (1940) in which expanding upon previous work made by Lessing centuries before, he seeks to trace the origin of modern abstract art which coincides, according to him, with the moment in which the various media began to become differentiated.

15. For example, Meadows (2011) highlights a synesthetic sensitivity present in Baudelaire, explaining how Baudelaire was more interested in exploring the connection between art and the senses rather than in the phenomenon of synesthesia.

16. Walter Gropius who designed the building and remained director of the Bauhaus until 1928. For an introduction to the Bauhaus see Curtis (1987, 309–316).

17. In his book Ways of Curating (2014) Hans Ulrich Obrist offers an insight into Harald Szeemann’s curatorial practice focusing on the 1983 exhibition The Tendency Towards the Total Work of Art which was a milestone in Obrist’s approach to curatorship and a landmark in other successive exhibitions by other curators such as Massimiliano Gioni who curated the 2013 Venice Biennale of Visual Arts under the theme of the Encyclopaedic Palace.

18. Passage from the interview I held with Johannes Deutsch in Wien in September 2012. The original interview was held in English and, if required, in German.

19. To give a couple of examples, the photographic portraits depicting Deutsch’s wife and son constitute the material icons for several of his works.

20. With Vision Mahler Deutsch begin to explore music and the interaction between sound and images in constant renewal. In the same line of exalting the intertwining between image, music and word, the multi-sensory opera Manfred is an interpretation by Deutsch of the romantic poem by Lord Byron put in music by Schumann. In Deutsch’s Manfred, the ever changing psychic landscape of the protagonist becomes central, with its ability to recall memories that then sink into obliviousness.

21. A short excerpt from Zeit Perlen can be accessed online with comments by the artist and other professionals involved. Available from: www.youtube.com/watch?v=zPcqpXZd4us (accessed February 2016).

22. While Peircian semiotics is commonly associated with logics, his phenomenology more carefully includes a bodily basis of semiosis:

23. From the interview to Deutsch.

24. The term “mental theatre” was coined by Lord Byron to describe his theatre plays that aim at addressing directly the minds of the audience/readers bypassing any physical constraints inbuilt in the actual representation on stage (Richardson 2003).

25. Van Campen devotes a few pages to the discussion of synesthesia as being a culturally-specific phenomenon. He briefly analyses the syneasthesia-based experience of the world in non Western cultures, for example among the Desana Indians in Colombia. See van Campen (2010, 101–102).

26. For an overview of contemporary art-science works see Wilson (2010).

27. From the exhibition catalogue Johannes Deutsch. Zwischen Erinnern und Vergessen. The translation in English from the original text in German is mine.

28. The term Pathosformel (pl. Pathosformeln) was coined by the art historian Aby Warburg in the early 1900s. With it Warburg meant to describe certain archetypal images that come back in different contexts throughout the centuries of art history. The different eras overlap as if they were sediments from different geological phases, allowing an image, which was absent for some time, to suddenly re-emerge from underground. The Pathosformeln are images that condense the original creation (pathos) with the repetitiveness of the canon to which they unintentionally refer (Formeln).