Balancing cohesion and tension in multimodal rhetoric. An interdisciplinary approach to the study of semiotic complexity

Abstract

What patterns of multimodal rhetoric characterizes the most successful texts meant for learning – inside or outside school? This article argues that the balance between cohesion and tension in the interplay between modes is an issue at the core of this challenging question. Texts meant for learning need both cohesion and tension in order to be both understandable and engaging. While cohesion has long been a central issue of interest among researchers of multimodal texts, the same can hardly be said about multimodal tension. The article offers a discussion of the concepts of multimodal cohesion and tension, relating them to three dimensions of user–text interaction: the dimensions of perception, interpretation and performativity. This elaboration concludes with the outline of a framework model for analytical purposes. The model is, thereafter, used in an illustrative analysis of an award-winning multimodal feature story collected from the online site of the New York Times.

Keywords:

• multimodal rhetoric
• cohesion
• genre
• digital discourse

Notes

For an elaboration on this observational claim, see, for example, Kress (2010) or Jewitt (2009).

By ‘informative texts’ I mean texts having the dissemination of knowledge as primary function, in contrast to texts having entertainment, explicit persuasion, aesthetic experience, etc. as primary functions.

For the description of such networks, see, for example, van Dijk (1980), Halliday and Hasan (1976) and de Beaugrande and Dressler (1981).

In Halliday's functional theory of language, cohesion represents one of three universal ‘meta functions’ inherent in a linguistic system, the other two being the experiental function (referring to the experienced world) and the relational function (reflecting and regulating the social relationship between language users) (Halliday 1985).

The three terms suggested here as representing basic types of cohesion can be traced back to the gestalt theories developed in the field of perceptional psychology in the 1920s (Wertheimer 1958, see also Bertin 1983).

The concepts of text and reading are generally used in a wide sense in this article.

The term rhetoric is in this article used in a wide and general sense, meaning the study of textual design seen in relation to communication goals, in line with the terminological practice of, for example, Kress (2010).

In a broader, hermeneutic perspective, we can relate this interest in cohesion also to Gadamer and his hermeneutic circle, focusing on the harmonizing interplay between the textual parts and the textual whole (Gadamer 1960).

Elaboration on semiotic provenance can be found in Kress and van Leeuwen (2001).

One of the most influential spokesmen for the belief in‘direct perception’, downgrading the significance of cognitive schemata in the process of perception, is Gibson (1979).

About modal density, see Norris (2004, 82).

See, for example, Panofski (1955), Barthes (1964), Mitchell (1994) and Kress and van Leeuwen (2006).

See, for example, Sacks, Jefferson, and Schegloff (1995), Halliday and Matthiessen (2004) and Fairclough (2010).

Influential representatives of performativity in critical and literary theory are Butler (1997) and Sedgwick (2003).

The complete text can be experienced on http://www.nytimes.com/packages/html/nyregion/1-in-8-million/index.html.

SND stands for the Society for News Design (www.snd.org).

See Manovich (2001) about the cultural shift from linear, narrative formats of films and books to the non-linear, user-driven forms of the database.

The analysis offered here – including the tables in Figures 3 and 4 – is of course not a registration of objective observations, but rather a result of a personal reading – as are all textual analyses going deeper than mere description. Nonetheless, given the criteria for analysis presented in the first sections of the article, there are reasons to assume that other analysts would come to conclusions not completely different from those given here.

See Kress and van Leeuwen (2001) about the interpretation of the human voice.