Gesturing towards theory

Karen Collins’ Playing with Sound presents an ambitious research project in a compact package. Moving quickly through chapters that begin by establishing an analytical orientation for ‘interacting with’ sound as opposed to ‘listening to’ it, and concluding with a survey of riches for future work, the book presents a central, innovative proposal: ‘playing with sound’ requires analyzing ‘kinesonic synchresis’. With this phrase, Collins aims to describe what is, at its most effective, a highly congruent temporal mapping of kinesthetic gesture with auditory playback. Her key example has been hiding in plain sight: returning repeatedly as pedagogical aid is Nintendo's Mario avatar, whose jumping movement Collins invokes to describe how and why the coordinated use of sonic material and player gesture can result in affectively powerful and semiotically meaningful experience when we interact with, rather than just listen to, sound in digital games.

‘Embodied cognition’ is the methodological orientation here; as Collins explains, ‘embodied cognitivist’ approaches depart from the understanding that ‘our cognitive processes use reactivations of sensory and motor states from our past experience’ (17). Collins nevertheless draws from widely ranging and at times competing theories of auditory and musical meaning. Michel Chion’s long-influential work on audiovisual ‘synchresis’ in cinema;¹ Rick Altman’s critical account of auditory interpellation in cinema;² idiosyncratic accounts of sonic experience (Murray Schafer’s ‘schizophonia’) or sound design (Walter Murch);³ Kiri Miller’s influential study of multilayered performance in digital games like Guitar Hero;⁴ phenomenologies of technology (Don Ihde) or of cinema (Vivian Sobchack);⁵ Roland Barthes’ ‘grain’;⁶ all in the interest of an embodied cognitivist account of player interaction with game sound.

As such, Playing with Sound is a highly synthetic, and often rewardingly ambitious, analysis of sound, embodiment, play, digitality, interactivity, and media combination and recombination. Collin's key proposal regarding ‘kinesonic synchresis’, presented in the first two chapters, is that it describes the emergence of affective and semiotic meaning in the player's gestural activation of auditory game material. Player action and auditory playback (or synthesis) give rise to emergent meaning and affective engagement that is, first, largely independent of the visual cues that may prompt and anchor such fusion on screen, and second, most powerful when gestural action and auditory playback are highly ‘congruent’ in duration. We would find this emergent, synchretic effect in just about any instance of interactive media that uses sound effectively, but it seems to me that Collins’ work, if implicitly, also proposes that we are not making optimal use of the greater potentials for meaning and engagement that can result when gesture and sound are designed to be congruent in the ‘peripersonal’ space of sound at the digital screen. This analysis of synchronized gesture and sound established in the first two chapters, the last three chapters go on to explore effects in gaming and in digital forms ancillary to it: identification and immersion via online game sound, including Voice-over-Internet-Protocol (VoIP); sound and music in performance in, say, machinema; and the reuse or modding of game sound in user-generated content or for artistic works.

The larger aims of Playing with Sound, with its emphasis on emergent, kinesthetic–sonic fusion (rather than audiovisual display), are prompted not simply by an interest in providing more apposite terms for interpreting the meanings and values of digital sound cultures, but also, arguably, by an interest in advocating greater design attention to the kinds of experiences in digital cultures that users-at-play find so compelling. Playing with Sound can be read as theorizing game sound not simply descriptively but also speculatively, that is, in support of further compositional experiment – and all the better to understand what precisely might enrich the production of emergent meaning and player engagement in digital games and the cultures of composition associated with them. It seems to me that the closely synchronized sound and gesture in which this book is most interested in may afford, in fact, highly musical meanings and effects, and crucially, beyond any specific musicological content of game music or sound design, and beyond any specifically musical virtuosity game players might contribute. What Collins calls kinesonic synchresis might conceivably both index and afford a kind of musical virtuosity in digital media cultures.

Yet Playing with Sound is perhaps compromised by its overweening prioritization of sound and gesture. The book suggests how and why gestural sound can be so potent, but nearly all of the examples discussed include visual streams whose meanings I would be hard pressed to redact from a definitive analysis of form or reception; but when visual figures are used, such a significant amount of graphical detail has been removed from their in-game appearance that the images are shorn of significant meaning. If there is considerable descriptive value in theorizing gestural sound as affording deep affective and semiotic engagement, as well as experimental value in forwarding gestural sound as an avenue for enhanced parsing or control of the flow of media streams as musical streams, these merits are undercut by our realization that inevitably, visual materials are also productive of meaning in just about all of the examples covered here. Whether responding to visual prompts to interact with sound, or requiring visual material to understand what gestural sound is doing in a game, the ‘player with sound’ may need to disambiguate ‘kinesonic’ material visually in the process of engaging with it ‘kinesonically’. So even where kinesonic meaning or affect may be dominant, complex displays of streaming visual material may yet be required to gesturally manipulate or interpret that fused, emergent material.

Of course, to demand attention to the visual dimension from a book that works strenuously to avoid doing just that is in large measure missing the point. More seriously, even within its own methodological contours, Playing with Sound's synthesis of highly disparate critical sources and methods is often distracting – and at times unconvincing. And ‘embodied cognition’ functions too often as a kind of generalized, analytical synecdoche, standing in for highly varied analytical problems never fully articulated. ‘Embodied cognition’ becomes a kind of ‘cognitive container’ into which just about any approach to analyzing sound, image, and gesture can be packed. Unfortunately, details of even previous work in embodied cognition (which has a longer history than Collins grants) not only do not get enough attention, but are badly mischaracterized. For example, Nicholas Cook's account of emergent meaning in musical multimedia is mentioned briefly;⁷ as the discussion zooms forward to elaborate ‘kinesonic congruence’, the cursory review of Cook's analysis of multisensory integration gets the names of his categories quite wrong, and mischaracterizes the way they work. Collins writes that Cook proposed a ‘contradict, contest, and contrast relationship of image and sound’ (26). Cook did propose a tripartite model of meaning arising from distinct media elements. But he understood sound and image as either ‘conformant’ (closely synchronized media streams producing meaning in similar terms); ‘complementary’ (less closely synchronized, each stream says something similar but in different terms); or finally, as ‘contesting’ one another (sound and image bring opposing meanings to one another). This mischaracterization is too bad, because deploying that kind of analytical schema might have produced a richer account, not simply ‘kinesonic congruence’, but also of kinesonically contesting, complementary, or conformant meanings and values.

And very often, discussions of previous work often feel imprecisely situated in the context of the book's methodological orientation. Readers conversant with the sources invoked throughout the book may feel that subsuming such a varied range of sources under the analytical resources of embodied cognitivism reduces the considerable complexity of many of them, to no particular gain. These problems may be as much questions about the manuscript review of Playing with Sound as about its production; the review process should have produced clarifications and corrections especially when previous work is mischaracterized.

Playing with Sound is an often exciting treatment of game sound offering several promising analytical inventions; but perhaps it functions better as a proposal of a theory of interactive game sound than a coherent theory of game sound in its own right. The overly synthetic nature of much of the discussion, and the mischaracterizations of previous work that appear along the way, may make Playing with Sound hard to use in many media or music studies contexts, and also make it a kind of ambivalent object for future work in the analysis or design of meaningful, affectively engaging interactive gesture affording new potentials for play and sociality.

Notes on contributor

James Tobias is Associate Professor in the Department of English at University of California Riverside, Riverside CA, United States, and the author of Sync: Stylistics of Hieroglyphic Time (2010, Temple University Press), a study of musicality in audiovisual media from early montage cinema to contemporary digital composition, and of numerous essays on musicality in time-based media. He is a member of the editorial boards of the journals Music, Sound, and the Moving Image, published by Liverpool University Press, and Games and Culture, published by Sage.

James Tobias
Department of English, University of California Riverside, Riverside CA, United States
[email protected]
© 2015 James Tobias
http://dx.doi.org/10.1080/20551940.2015.1079976

Notes

1. Chion, Audio-Vision, 28.

2. Altman, “Sound Space,” 61–62.

3. Schafer, The New Soundscape, 91; Murch, “Dense Clarity Clear Density.”

4. Miller, Playing Along, 150.

5. Ihde, Bodies in Technology; Sobchack, Carnal Thoughts.

6. Barthes, “The Grain of the Voice,” 179–189.

7. Cook, Analyzing Musical Multimedia, 98–106.