Visuo-spatial construals that aid in understanding activity in visual-centred narrative

ABSTRACT

Cohn (2016) posits two constraints critical to understanding sequential static images as narrative – the continuity constraint, assuring that elements in one frame are coreferential with elements in other frames, and the activity constraint, assuring that states in each frame are related with respect to passing of time, shifting of vantage point, or other factors relevant to activity. We identify seven construals used in ordinary human understanding of the physical world that help account for how viewers interpret activity in static visual narratives and that we propose form the foundation of the activity constraint. In an elicitation experiment, these construals also supported understanding of depicted activity in dynamic storytelling in Irish Sign Language, suggesting that a range of action information in sign language narratives is conveyed via reliance on visuo-spatial construals independent of language.

KEYWORDS:

• Sign languages
• visual narrative
• depiction
• iconicity
• construals about activity

Acknowledgements

Donna Jo thanks the Trinity Long Room Hub for their support. The initial discussions that led to this work emerged as a result of her TLRH Fellowship at Trinity College Dublin in 2012. Lorraine thanks Swarthmore College for their support. Much of the background reading and discussion of this work took place while she served as the Julian and Virginia Cornell Distinguished Visiting Professor in 2013–14. Lorraine would also like to acknowledge the input of Centre for Deaf Studies Junior Sophister students who worked with the Frog, where are you? data in 2016–17, selecting some of the examples that are included here. Thank you to Loredana Macari, Edel Walsh, and Ger Brennan. Both of us thank the Irish Sign Language users who kindly contributed to the Signs of Ireland Corpus and to the elicitation task later recorded. Research Ethics approval for the data collection was secured in 2013 from the School of Linguistic, Speech and Communication Sciences, Trinity College Dublin and from the Institutional Review Board at Swarthmore College in Pennsylvania, USA. We are indebted to the heroic efforts of the anonymous reviewers of earlier drafts of this work and the patience and guidance of the journal editor and to Patricia Irwin for helpful discussion.

Disclosure statement

No potential conflict of interest was reported by the author(s).

ORCID

Donna Jo Napoli http://orcid.org/0000-0001-6320-1086

Lorraine Leeson http://orcid.org/0000-0002-0887-7275

Notes

1 For example, groups that rely on uniform interpretation of images (engineers, architects, surgeons …) collectively establish conventions for producing and interpreting those images, using systems that “are both conventional crystallizations of visual perception and extensions of it” (Greenberg, 2011, p. 31). These conventions must be as stable as the interpretive conventions used in language. However, groups might choose different conventions, while still realising their goals. Streeck (2008, p. 287), for example, talks of a “culture-bound repertoire” that artists share, such as one system of drawing using perspective where another doesn’t.

2 Additionally, of course, sequencing in visual narrative can signal causality (Hanfmann, 1957).

3 The perception of iconicity in any particular sign depends to a large extent on an individual’s language and socio-cultural experience (Occhino et al., 2017, p. 104; Pizzuto & Volterra, 2000; Taub, 2001; Wilcox, 2000); it is not to be taken as an objective property of a sign.

4 Debates over the theoretical status of structures that rely on visuo-spatial construals for their interpretation abound. Classifier predicates are a prime example. They have been argued to be non-linguistic, belonging to a system of visual representation that is part of the larger cognitive ability of human communication (Cogill-Koetz, 2000a, 2000b). They have been argued to be a blend of gesture and linguistic information, specifically, morphological information (Schembri et al., 2005). And they have been argued to be fully linguistic, based on their phonological and syntactic properties (Benedicto & Brentari, 2004). The same range of conclusions can be found for agreeing verbs (for an overview, see Meier, 2002). We refer the interested reader to Braun et al. (2001), a study showing that the general networks recruited for the production of narratives in sign and spoken languages are the same, and to Blanco-Elorrieta et al. (2018), a study showing that the same neurobiology is responsible for the planning of structured linguistic expressions in sign and spoken languages.

5 For example, an eye-tracking study shows that when comics are displayed one frame at a time, people spend more time on each frame than when an entire sequence is presented at once (Foulsham et al. 2016). This may be due to the fact that the entire sequence allows for regression (where the viewer can look back, typically at the immediately preceding frame) to check information, while the one-frame-at-a-time order precludes regression and thus demands that the viewer glean all the relevant information before proceeding to the next frame.

6 It’s valuable to note, however, that in some traditions, watching the process of drawing may itself be meaningful, and the end result may, in fact, be transitory. For example, in Central Australia’s desert-dwelling Arrente community, sand drawings can be accompanied by speech, song, gesture, and sign language (Wilkins, 2016; Green, 2014; for discussion of this ritualistic sign language see Strehlow, 1978). The result is a narrative that none of those media alone could construct; considering the visual parts of the narrative, the sand offers an aerial (bird’s eye) perspective that complements the sign perspective. The narrator smooths space on the sand in front of her and draws and signs alternately, exploiting the 3-dimensional opportunities of the sand in various ways, including making deeper impressions in the sand to represent heavier characters in the narrative, or retracing figures several times, making the impressions deeper, to represent the passing of time of that figure in that place. Like signing, the sand drawings rely on cross-modal metaphor (Napoli, 2017).

7 This harks to research on human infants that shows their perception of objects relies more heavily on the motion of the object than on a general tendency to maximise figural goodness (Spelke, 1990, 1991, 2010).

8 The literature seems to use the terms entity and object interchangeably. We have chosen to use only object in our own terminology, for consistency’s sake.

9 For example, recently, work from a cognitive/ functional perspective, analysing corpora in Irish Sign Language and American Sign Language, suggests that pragmatic inferencing is a mechanism for pronoun grammaticalization in discourse (Leeson et al. 2013). Over time, pronominal referencing increases in complexity due to factors such as shifting point of view. Leeson and colleagues demonstrate how embodiment promotes inferencing in complex clausal constructions. In line with this work is a recent study by Frederiksen and Mayberry (2016) on referent tracking in ASL discourse. They conclude that there is a hierarchy among referential expression types with respect to which type is used depending on accessibility of the referent from discourse context. They suggest that depiction (particularly role-shift) might be used in sign languages in contexts where pronouns are used in spoken language (and see further complexities of reference in Hodge et al., 2019, in a study of Auslan, the sign language of Australia).

10 The most extensive study using Frog, where are you? is Ferrara (2012, on Auslan), on depiction types and combinations in narratives, mentioning some spatial details, but focusing on different issues from ours. While there is no discussion of relative spatial relationships of objects in one scene as compared to others (our key concern here), Ferrara (2012, p. 207 ff.) shows that sequencing of scenes reflects chronological timing.