A multimodal analysis of enactment in everyday interaction in people with aphasia

ABSTRACT

Background: “Multimodal communication” is a relatively common term in aphasia research. However, the scope of studies on multimodal interaction in aphasia is generally restricted to one or two multimodal resources, and the type of discourse analysed is often not representative of authentic interaction. Finally, the interpersonal (versus referential) functions of multimodal resources are frequently overlooked.

Aims: The purpose of this study was to explore the multimodal realisation of enactments by people with aphasia in everyday interaction.

Methods & Procedures: Authentic interactions of six people with aphasia interacting with communication partners of their choice were systematically analysed. Frameworks originating from non-brain-damaged studies were applied to examine the characteristics and functions of linguistic, multimodal, and stance-taking resources used to realise enactments.

Outcomes & Results: Even though the participants used the same multimodal resources as non-brain-damaged communicators, the frequencies and characteristics were different. The relationship between multimodal resources and interpersonal functions was different as well.

Conclusions: People with aphasia use the same multimodal resources as non-brain-damaged communicators, indicating their retained strengths. However, their higher use of intonation, gesture, and – to a lesser extent – facial expression indicates that these may be important “meaning making” resources for them, which could be utilised more in therapeutic endeavours.

KEYWORDS:

• Aphasia
• discourse analysis
• multimodality
• everyday interaction

Introduction

The use of “non-verbal” resources has received considerable attention in aphasia research. Previous studies have examined the forms, potential and/or utility of gestures (e.g., Hogrefe, Ziegler, Weidinger, & Goldenberg, 2017; Pritchard, Dipper, Morgan, & Cocks, 2015; Rose, 2006; van Nispen, van de Sandt-Koenderman, Sekine, Krahmer, & Rose, 2017); pantomime (e.g., Nispen, Sandt-Koenderman, Mol, & Krahmer, 2014) and pointing (e.g., Klippi, 2015). Even though these studies have provided invaluable insights into communication abilities other than spoken language in aphasia, many other modalities such as gaze, facial expression, and posture, have often been overlooked. Studies resulting from Conversation Analysis principles, on the other hand, do acknowledge the importance of, for example, intonation, facial expression (e.g., Fromm et al., 2011; Laakso, 2014; Lind, 2002), and “body language” (Fromm et al., 2011, p. 1434). However, the frameworks applied in these studies tend to be of a more descriptive nature because of their qualitative, data-driven approach, where research questions originate from the data itself rather than being posed prior to carrying out the analysis. In aphasia therapy, “multi-modal therapy” – involving either compensation techniques when spoken communication fails to be restored, or facilitation technique to re-establish language and speech – usually refers to the use of drawing, gesture, reading and writing (e.g., Rose, Attard, Mok, Lanyon, & Foster, 2013; Rose, Mok, Carragher, Katthagen, & Attard, 2016). However, even though the term is used frequently in the aphasia literature, there is no consensus regarding its definition either (Pierce, O’Halloran, Togher, & Rose, 2018).

Another striking characteristic of studies assessing multimodal communication in aphasia that have been carried out so far is their approach to function. All communication is, and has always been, multimodal (Kress & Van Leeuwen, 1996). Therefore, analyses focused solely or primarily on language cannot adequately account for meaning in interaction (Jewitt, 2014). However, the traditional opposition of “verbal” and “non-verbal” communication in aphasia research presumes that the verbal is primary and therefore the “most important” (Jewitt, Bezemer, & O’Halloran, 2016). Just like in non-brain-damaged (NBD) research the point of reference and focus of analysis in aphasia research has traditionally hinged on speech, its central units usually being linguistic units (e.g., “intonation units”) or units defined in linguistic terms (e.g., a “turn” is defined in terms of “who is speaking”) (Bezemer & Jewitt, 2010). As such, modes of communication other than language are considered accompaniment, support, or even substitutes for language rather than modes which are crucial to understand communication.

This is clearly illustrated by Kong, Law, and Chak (2017) discussion of the “functional roles” (p. 2032) of gestures in communication. Kong et al. (2017) distinguish eight functions of gestures, all of which reflect their contribution to the semantic content of communication (e.g., providing “additional information to the message being conveyed”, “enhancing speech content”, “providing alternative means of communication” (p. 2032, our italics)); gestures are considered helpers to speech. Interpersonal functions – referring to the fact that speakers not only talk about something but are always talking to and with others with a particular perspective or stance in mind – are considered “nonspecific” or “noncommunicative” (p. 2032). This overview of potential functions of gestures in interaction involving people with aphasia (PWA) demonstrates how little is known about the role multimodal communication plays in aphasia. As stressed by Adami (2016) and Jewitt (2014), the examination of relations among modes is considered key to understanding any instance of communication.

The current study

The current study was designed to explore the relations among interactional modes in everyday interactions involving PWA. To do so, a discourse phenomenon in which multimodality has been argued to play a crucial role was assessed. Various terms have been introduced to refer to this phenomenon, such as demonstration (Clark & Gerrig, 1990), constructed dialogue (Debras, 2015), fictive interaction (Stec, 2016), and enactment (Groenewold & Armstrong, 2018; Kindell, Sage, Keady, & Wilkinson, 2013; Wilkinson, Beeke, & Maxim, 2010). Since enactment best reflects the multimodal nature of the phenomenon under study, this term will be applied here.

Enactment

When enacting, communicators depict to recipients aspects of a reported scene or event by employing direct reported speech and/or other behaviour such as gesture, body movement and/or prosody (Goodwin, 1990; Streeck & Knapp, 1992; Wilkinson et al., 2010). As such, enactment provides clear opportunities for multimodal communication. Previous research has shown that in PWA the use of enactment is not only preserved (Hengst, Frame, Neuman-Stritzel, & Gannaway, 2005; Ulatowska & Olness, 2003; Ulatowska, Reyes, Santos, & Worle, 2011; Wilkinson et al., 2010) but also increased (Berko Gleason et al., 1980; Groenewold, Bastiaanse, & Huiskes, 2013). One of the candidate explanations for this increase is that enactment is usually heavily marked with paralinguistic and non-linguistic behaviours, allowing the PWA to add information to talk that would otherwise be too complex to put into words (Groenewold & Armstrong, 2018; Günthner, 1999; Hengst et al., 2005). In other words: enactment could be a complementary or even compensatory device for PWA (Groenewold, 2015; Wilkinson et al., 2010). The current study will further explore the characteristics of this interactional resource for people with aphasia.

In Transcript 1, an example is provided of a PWA (H, all initials are pseudonyms) using an enactment to explain to his friend (F) how other members of an association used to look at him when he struggled with his language.

Transcript 1:

1.   H:     and then eh

2.          and then eh

3.          well eh ((mimicks facial expression))

4:  F:      they look [at you]

5:  H:     [yes yes yes]

Enactments, by their very nature, are interpersonal: There is a point to a person presenting the information in a certain way that provides the enacted/enacting communicator’s stance on that information. Conveying evaluative, modalising meanings rather than referential content (Olness & Englebretson, 2011), enactment goes beyond using multimodal resources for concrete compensatory information purposes only.

Since the importance of multimodal resources – and the way in which they are mutually combined – has been described more extensively in the typical interaction literature, frameworks originating from studies into non-brain-damaged (NBD) interaction will be applied. Below the characteristics that have been shown to be of relevance in the study of enactment in aphasic and/or NBD interaction will be discussed. Attention will be paid to the linguistic characteristics, the multimodal characteristics, and the relationship between the multimodal realisation and the interpersonal functions of enactments.

Linguistic characteristics of enactment

Previous research showed that enactments produced by PWA are often notable in terms of the distinctive grammatical practices within which they are produced (Wilkinson et al., 2010). More specifically, people with agrammatic aphasia exhibit a preference for enactments without a reporting verb and/or a person reference (“bare” enactments, e.g., Transcript 1). Such enactments are grammatically relatively easy to produce (Groenewold et al., 2013). Person references (i.e., a personal pronoun or a name), and reporting verbs (e.g., say, be like, whisper, shout, etc.) can be present or absent (independently of each other). Another characteristic is the question of who is being enacted (e.g., self, addressee, absent third party, prototypical person, etc.).

Multimodal characteristics of enactment

Stec (2016) identified five multimodal articulators (i.e., intonation, gesture, facial expression, speaker gaze, and body posture) that play a role in the realisation of enactments by NBD communicators. Relevant literature on these articulators is discussed below. The categorisation system will be further discussed in the Methods section.

Intonation

Even though intonation is a topic that has not been addressed systematically yet in the aphasia literature, some studies have acknowledged its importance. Disturbed prosody, for example, has been identified as one of the characteristics of agrammatism (e.g., Seddoh, 2004). Furthermore, Goodwin (2010) described a man with a vocabulary of only three words who could combine lexico-syntactic structure with prosody in such a way that a whole that is greater than any of its parts was created. In a Norwegian case study, Lind (2002) showed that prosody can play an important role in the pragmatic contextualisation of direct reported speech. Altogether these studies demonstrated that prosody can play an important role in meaning-making in communication in PWA.

Gesture

Despite the broad potential relevance of gesture in aphasia, the topic is often claimed to be understudied (see Linnik, Bastiaanse, & Höhle, 2016 for an overview). Whereas most researchers agree that the processes underlying gesture and language production are shared or closely related (Dipper, Cocks, Rowe, & Morgan, 2011; Goodwin, 2000; Mol, Krahmer, & van de Sandt-Koenderman, 2013), others suggest that the gesture system can remain functional even when language production is severely impaired (e.g., Akhavan, Göksun, & Nozari, 2018). It has been shown that a great proportion of gestures produced by most PWA is crucial for understanding their communication (van Nispen et al., 2017). Strikingly, even though gestures play an important role in everyday communication (Kendon, 1997), most studies into the role of gestures in communication involving PWA have relied on discourse elicitation tasks such as semi-structured conversations (e.g., van Nispen et al., 2017), procedural discourses (e.g., Pritchard et al., 2015), responses to a communicative scenario (e.g., Mol et al., 2013), video clip retellings (e.g., Hogrefe, Ziegler, Wiesmayer, Weidinger, & Goldenberg, 2013), story retell samples (e.g., Sekine & Rose, 2013), cartoon descriptions (e.g., Dipper et al., 2011) or personal narratives (e.g., Sekine, Rose, Foster, Attard, & Lanyon, 2013) rather than authentic interactions. As a consequence, the meaning-making potential of gestures in conversations involving PWA remains understudied. All studies suggest, however, that gestures play an important role in meaning-making in individuals with aphasia.

Facial expression

Little is known about facial expression in aphasia since only a few studies on this topic have been carried out. Furthermore, when assessing facial expression in aphasia there has been a main focus on perception (e.g., Duffy & Watkins, 1984; Feyereisen & Seron, 1982) rather than production (but see, e.g., Buck & Duffy, 1980; Duffy & Buck, 1979). In addition, the way “spontaneous” nonverbal expressiveness in the production studies was elicited was through responses to different types of affective slides (i.e., familiar people, pleasant landscapes, unpleasant scenes, and strange photographic effects). These latter studies suggested that PWA are equal to or more expressive than control participants (Buck & Duffy, 1980; Duffy & Buck, 1979).

Speaker gaze

In exploring topics such as word-searching behaviour (Laakso & Klippi, 1999), the role of gaze in interaction involving PWA has received some attention. For example, through applying Conversation Analysis, research has demonstrated that PWA use gaze for interaction management, such as turning gaze to the recipient to solicit assistance from the recipient (Damico, Oelschlaeger, & Simmons-Mackie, 1999; Goodwin & Goodwin, 1986; Wilkinson, 2007). Furthermore, PWA have been argued to use shifts in gaze to hold or yield a turn in conversation (Laakso, 2014). Such gaze practices, and also those of withdrawing gaze to display a word search as one’s own, self-directed activity, are similar to those used by NBD communicators (e.g., Laakso, 1997).

In the organisation of enactments in NBD communicators, gaze is argued to play a crucial role. At the start of enactments, NBD communicators often direct their gaze away from the listeners, indicating that they are entering into a part of the story that will be enacted rather than narrated (Sidnell, 2006).

Body posture

Body posture is another topic that has received little attention in aphasia research. An exception to this is a study carried out by Laakso (2014), who examined how PWA display affect in conversation. She argues that shifts in body postures are one of the most common affect displays and that PWA use affect displays in close coordination with shifts in body posture that reflect turn organisation (Laakso, 2014). This finding resembles an observation by Bloch and Beeke (2008), who described how a PWA used body posture – in combination with gaze and gesture – to project something of the meaning of a turn not only before and during, but also after its verbal content.

Apart from the different forms of multimodality and their reciprocal relations, their relationship with stance-taking has been described in the NBD enactment literature (e.g., Debras, 2015). Stance is “a display of socially recognized point of view or attitude” (Ochs, 1993, p. 288). Debras (2015) argues that, when enacting, NBD communicators use changes in voice pitch and pantomime when they distance themselves from a stance attributed to the enacted person. When communicators endorse a stance attributed to an absent third party by enactment, there is continuity in gesturing style and intonation (Debras, 2015).

The current study aims to address the issues raised above in relation to linguistic and multimodal patterns of enactment in PWA through analysis of authentic interactions, asking the following questions:

1. What are the linguistic and multimodal characteristics of enactments produced by PWA in everyday interaction?

2. Is there a relationship between the amount of linguistic information and the number of multimodal articulators (intonation, gesture, facial expression, gaze, posture change)?

3. To what extent do the characteristics of these enactments resemble findings for NBD communicators?

   1. To what extent are the qualitative characteristics similar?

   2. To what extent are the quantitative characteristics similar?

   3. To what extent is the relationship between stance and shifts in intonation and/or gesturing style similar?

Methods

Enactments occurring in a corpus of authentic, everyday interactions collected by the first author in The Netherlands in 2012 were analysed. In this section, an overview of the corpus collection and annotation procedures is provided.

Corpus

The corpus consists of approximately 8 h of data and comprises 18 videos ranging in length from 6 min to 1 h and 22 min (M = 26 min, SD = 22 min) and 112 enactments (M = 18.7, SD = 9.4) produced by PWA. Six people with chronic aphasia (>6 months post-onset) were recorded talking to conversation partners of their choice (see Table 1).

Table 1. Participant characteristics.

P	Gender	Age (years)	BDAE severity rating	Hemiparesis (right-sided)	Videos analysed (n)	Conversation partners	Total duration videos (minutes)	Enactments produced (n)
1	Male	57	1	Yes	3	Friend (n= 3)	126	14
2	Male	64	1	Yes	4	Son (3 times); daughter	38	12
3	Male	62	3	No	4	Colleague (twice), friend (n= 2)	62	12
4	Female	66	2	No	4	Husband (twice); daughter (twice)	115	15
5	Female	38	2	Yes	2	Husband, friend	100	23
6	Female	57	1	Yes	1	Sister-in-law	41	36

Their scores for the Boston Diagnostic Aphasia Examination (third edition, BDAE-3, Goodglass, Kaplan, & Barresi, 2001) aphasia severity rating scale ranged from 1 (“All communication is through fragmentary expression; great need for inference, questioning, and guessing by the listener. The range of information that can be exchanged is limited, and the listener carries the burden of communication”) to 3 (“The patient can discuss almost all everyday problems with little or no assistance. However, reduction of speech and/or comprehension make conversation about certain material difficult or impossible”) (M = 1.67, SD = 0.82). In order to provide more insight into the PWA’s speech characteristics, the aspects of the BDAE-3 rating scale profile of speech characteristics (Goodglass et al., 2001) that could be assessed based on the interactional data (i.e., all but repetition and auditory comprehension) were rated. These ratings are presented in Table 2.

Table 2. PWA’s speech characteristic scores for the BDAE-3 rating scale profile of speech characteristics (Goodglass et al., 2001) that could be assessed based on the interactional data. P: participant. Note that Paraphasia in running speech is only rated if phrase length is 4 or more (Goodglass et al., 2001).

	Speech characteristic
	Articulatory agility	Phrase length	Grammatical form	Melodic line	Paraphasia in running speech	Word finding relative to fluency
Participant	(1: Unable to form speech sounds, 7: Never impaired)	(1: 1 word, 7: 7 words)	(1: No syntactic word groupings, 7: Normal range of syntax, normal facility with grammatical words)	(1: Word-by-word or aprosoidc, 7: Normal prosody)	(1: Present in every utterance, 7: Absent)	(1: Fluent but empty speech, 7: Output primarily content words)
P1	3	2	2	3	NA	6
P2	2	3	3	3	NA	6
P3	2	7	7	7	2	4
P4	2	7	4	1	6	7
P5	2	3	2	4	NA	5
P6	4	3	2	4	NA	5

All participants completed an informed consent procedure, involving an oral and written aphasia-friendly explanation of the goal and nature of the research allowing for an informed decision about participation, and several options regarding the use of the collected materials. All participants agreed to participate voluntarily and granted permission to use written transcripts of the recordings for scientific publications, and video-recordings for scientific conferences and education purposes.

Participants were asked to make video recordings of conversational activities, representative of types of activities that would occur if the video equipment were absent or if requests for data had not been made. The first author visited the participants at home to provide the camera, explain and complete the informed consent procedure, and explain how to position and operate the camera. No schedules or topics were predetermined, and video recording occurred at the participants’ discretion. Even though the participants were invited to record dialogues, since the purpose of the study was to examine enactment as it occurs in authentic everyday interaction, multi-party interactions (n = 5) were included for analysis as well. The remaining 13 conversations were two-party interactions, representing casual conversations between the PWAs and a friend (n = 5), child (n= 4), spouse (n = 3), and sister-in-law (n = 1).

Analysis

The first author identified, transcribed, and coded all enactments. Around 20% of the identified enactments were re-coded by an independent rater, who was familiar with the notions of enactment and its multimodal characteristics. Enactments were identified based on the presence of quoting predicates such as say or be like, or in the case of bare quotes, a shift in indexicals (e.g., personal pronouns, demonstratives, deictics, time reference), or prosodic or non-verbal markers, such as the occurrence of pauses in speech, shifts in posture, gaze, facial expression, voice quality and pitch height (see also Groenewold, Bastiaanse, Nickels, Wieling, & Huiskes, 2014; Lind, 2002).

The scheme for analysis is based on annotation schemes developed by Stec (2016) and Debras (2015) (see Table 3). It contains variables for linguistic features pertaining to enactment, multimodal features which contribute to enactment (Stec, 2016), and labels for stance-taking (Debras, 2015).

Table 3. Scheme for analysis used for this study (adapted from Stec (2016) and Debras (2015)).

Category	Variable	Values
Linguistic information	Enacted character	Self Addressee Absent third party Generic/prototypical Inanimate/animal Multiple
	Person reference	Present Absent
	Reporting verb	Present Absent
Multimodal resources	Character intonation	Present NA Unclear Absent
	Gesture	Character viewpoint Other Absent
	Character facial expression	Present NA (e.g., inanimate) Unclear Absent
	Gaze	Maintained with addressee Away from addressee Late change away from the addressee Late change towards the addressee Quick shift Unclear
	Posture change	Horizontal Vertical Sagittal Unclear None
Stance-taking	Affiliation with enacted character	Affiliation Disaffiliation NA Unclear

Linguistic realisation of enactment

First, the linguistic realisation of enactment was analysed. Based on the person reference (or local context in case of absence of a person reference), the enacted character was annotated. A communicator can enact him/herself, but also the addressee, or someone who is absent. Furthermore, a communicator can enact a generic or prototypical person (e.g., one), an animal or non-animate object (e.g., the “sound” of a needle in Transcript 2, line 6), or multiple persons (e.g., Mary and Pete in Transcript 3, line 10).

Transcript 2: ZZLLP

1.  K:  me (.) sewing (.) Mary?

2.  M:  well I (.) I can do it you’ve got [eh]

3.  K:   [yeah:]

4.  M:  I am good at (.) shortening and eh but it takes long (.) and nowadays

5.     [with those] (.) new machines

6.  K:  [zzlp]

7.     (1.0)

8.  M:  then you do double needles

Transcript 3

1.  N:    I said go there

2.  V:    yes of course, no problem

3.  N:    and

4.  V:    of course

5.  N:    yes but eh Mary was here

6.  V:    yes

7.  N:    and Pete

8.  V:    hmm

9.  N:    and eh no

10.  N:    eh said no then you said eh come then

11.  N:    well and well hm hm hm

12.  N:    eh I said go there

13.  V:    of course, no problem

Next, the absence or presence of an explicit person reference (i.e., name or pronoun) and reporting verb (e.g., say, be like) was coded.

Multimodal resources used to realise enactment

After coding the linguistic characteristics, the multimodal characteristics of the enactments were coded.

For character intonation, it was noted whether the communicator used special intonation which was noticeably different from the communicator’s narrative voice, e.g., a change in pitch height, duration, intensity, or voice quality, to indicate aspects of the enacted character’s speech. In some cases (e.g., onomatopoeia), there is no character whose intonation can be enacted (e.g., the needle “sound” in Transcript 2). Rather than annotating their character intonation as absent, we added a new category NA to mark these instances.

For gesture (Hands in Stec (2016)) a distinction was made between character viewpoint gesture (communicator demonstrates a gesture performed by the enacted character), other gesture (gestures which do not reflect a gesture performed by the enacted character), and no gesture.

Character facial expression could be present (the communicator’s facial expression changes to demonstrate the enacted character), absent, or unclear. Just as for character intonation, NA was added to mark instances for which mimicking of facial expression is not possible (e.g., objects).

For gaze, the coding system applied by Stec (2016) was used, complemented with the value late change towards addressee, indicating the communicator’s gaze moves towards the addressee after the enactment started (see Transcript 4).

Transcript 4: Hi-jack!

1.  H:   one shou– one should never eh ((gazes away from D))

2.  H:   say eh: ((shifts gaze towards D))

3.  H:   pilot eh hi Jack! ((gazes at D))

4.  H:    ((laughs)) ((gazes at D))

5.  D:   ((laughs))

The final multimodal resource described by Stec (2016), posture change, was coded without modifications or additions. This variable indicates the direction of movement or shift in body orientation made by communicators during the enactment and may reflect movements made by the hands, torso and/or hands. All movements which were not self-adaptors (e.g., re-adjusting seated position) were analysed. The values for posture change were horizontal, vertical, sagittal (from back to front or vice versa), unclear, and none.

For an illustration of gesture, posture change, and facial expression, consider Transcript 5. This is taken from one of the interactions between B, a person with aphasia, and A, B’s sister-in-law. In this excerpt, head movements, character viewpoint gestures, and facial expression are used to enact Hannah, one of the aphasia centre volunteers.

Transcript 5: Cleaning

1.  A:    so who does the vacuuming cleaning there?

2.  B:    eh cleaning lady

3.  A:    right

4.  B:    cleaning lady

       (2.7)

5.  B:    no eh Hannah eh ((raises hand))

6.  B:    ((raises eyebrows, tilts head back, moves

7.      horizontally, short waves))

8.  B:    ((laughs))

9.  A:    well I think she’s right in that she says

10.  A:    I’m not cleaning the toilets

Number of articulators used to realise enactment

Following Stec (2016), based on the multimodal resources annotations, a variable called articulator count was created which counts the number of multimodal articulators (resources) pertaining to enactments. It ranges from 0 to 5 to indicate no articulators (0) through all articulators (5). For example, an enactment with character intonation (1), no gesture (0), character facial expression (1), gaze change towards addressee (1), and sagittal posture change (1) would be counted as 4.

‘Meaningful’ use of multimodal resources

For the assessment of meaningful use of multimodal resources the procedures and interpretations suggested by Stec (2016) were followed. She suggested that meaningful use of multimodal resources is indicated by change. Thus, away from addressee, quick shift and late change are active uses of gaze while maintaining gaze with the addressee is not. Similarly, for character intonation, it was noted whether the communicator used special intonation which was noticeably different from the communicator’s normal or narrative voice (e.g., a change in pitch, volume, accent, rate). Cases labelled unclear or NA (e.g., a completely non-verbal enactment) were counted as not active.

(Dis)affiliation

Finally, based on the analysis of the sequential context, the raters indicated whether the communicator agreed (affiliation) or disagreed (disaffiliation) with the enacted character (Debras, 2015). As in the case of character intonation and character facial expression, an extra value NA was added to the coding scheme to categorise those instances where the predetermined values are not possible (e.g., when a hypothetical person is enacted).

IRR reliability

The corpus consisted of 112 enactments in total. All enactments were coded by the first author. Eighteen percent (20 enactments) of the data was compared with annotations made by an independent rater. Cohen’s κ was run to determine if there was agreement between the two raters on the nine variables. For two of the linguistic variables (enacted character and person reference), there was a perfect agreement between the two raters, κ = 1.00 (p < 0.001). There was an almost perfect agreement for the third linguistic variable, reporting verb, κ = 0.806 (95% CI, 0.547 to 1.00), p < 0.001. For character intonation Cohen’s κ could not be calculated because one of the rater’s ratings had no variation. The percentage agreement for this variable was 0.95. There was substantial agreement for gesture, κ = 0.751 (95% CI, 0.508 to 0.994), p < 0.001, character facial expression, κ = 0.645 (95% CI, 0.273 to 0.784), p= 0.01, and gaze, κ = 0.745 (95% CI, 0.525 to 0.965). The inter-rater agreement for posture change was almost perfect, κ = 0.851 (95% CI, 0.663 to 1.000), p < 0.001. Finally, the agreement for affiliation was almost perfect as well, κ = 0.847 (0.655 to 1.000), p< 0.001.

Results

In this section, the linguistic and multimodal characteristics for enactments produced by the PWA are reported. Second, the relationship between the linguistic characteristics and the number of multimodal articulators is presented. Finally, a comparison is drawn between the current findings and those previously reported for NBD communicators.

Linguistic and multimodal characteristics for PWA

The PWA most frequently enacted other people (39.3%) or themselves (38.4%) (Table 4). However, they also regularly enacted multiple characters (e.g., “and then they said, alright, let’s go then” (participant 5)), and generic or prototypical characters (e.g., “on a plane one should never say, Hi Jack! Hi-jack!” (participant 2)).

Table 4. Enacted characters.

Enacted character	n	%
Self	43	38.4
Addressee	1	0.9
Absent third party	44	39.3
Generic/prototypical	9	8.0
Inanimate/animal	3	2.7
Unspecified	2	1.8
Multiple	10	8.9

Total of 56.3% of all produced enactments were so-called bare enactments (i.e., no person reference and no reporting verb, e.g., “and then eh, well eh ((mimics facial expression of group of people))”) (Table 5). Approximately a fifth of the enactments were marked by a person reference and a reporting verb (e.g., I said, “buy ticket”). Nearly another fifth of the enactments were introduced by a person reference only (e.g., I, “sewing, Mary?!”). Finally, enactments introduced by a reporting verb only (e.g., then, said, “hey, man!”) occurred less frequently (n= 5, 4.5%).

Table 5. Person reference and reporting verbs in enactments.

Quoting predicate	n	%
None	63	56.3
Only person reference	21	18.8
Only reporting verb	5	4.5
Person reference and reporting verb	23	20.5

The multimodal resources used to realise enactments are presented in Table 6. This overview contains the results for all enactments, regardless of whether or not they were linguistically marked. The participants enacted the enacted characters’ intonation in almost 80% of all instances. In some cases (8%) enactment of character intonation was coded as NA because the type of enactment did not allow for intonation enactment, such as instances which consisted exclusively of movement and/or facial expression.

Table 6. Multimodal resources used to realise enactments.

Resource	Value	n	%
Character intonation	Present	87	77.7
	NA	9	8.0
	Unclear	2	1.8
	Absent	14	12.5
Gesture	Character viewpoint	42	37.5
	Other	16	14.3
	Absent	54	48.2
Character facial expression	Present	58	51.8
	NA	1	0.9
	Unclear	12	10.7
	Absent	41	36.6
Gaze	Maintained with addressee	36	32.1
	Away from addressee	28	25.0
	Late change away from addressee	6	5.4
	Late change towards addressee	21	18.8
	Quick shift	18	16.1
	Unclear	3	2.7
Posture change	Horizontal	20	17.9
	Vertical	6	5.4
	Sagittal	23	20.5
	Unclear	0	0
	None	63	56.3

Nearly half of the enactments (48.2%) were not accompanied by a gesture. PWA produced a higher percentage of character viewpoint gestures (37.5%) than gestures which do not reflect a gesture performed by the enacted character (14.3%).

Character facial expression was enacted in more than half of all enactments (51.8%). In some cases (10.7%) this variable was coded as unclear because the participant’s facial expression was not clearly visible or the face was turned away.

The most frequently occurring value for gaze during enactment was maintained with addressee (32.1%), followed by away from addressee (25.0%). In nearly 20% of the cases, gaze was coded as late change towards addressee, which did not occur in Stec (2016)’s categorisation system.

In more than half of the cases, enactments were not accompanied by a posture change. In the case of posture changes, it was most frequently a sagittal movement (20.5%), followed by a horizontal movement (17.9%).

Relationship between linguistic characteristics and number of multimodal articulators

In Table 7, the relationship between the absence or presence of linguistic markers (person reference and/or reporting verb) and the number of multimodal articulators used to realise enactment is presented. Bare enactments were most often accompanied by three active multimodal articulators (33.33%). The same goes for enactments that were preceded by only a person reference. Enactments that were preceded by only a reporting verb often co-occurred with three (40.0%) or four (40.0%) multimodal articulators. The most frequently occurring number of active multimodal articulators for enactments preceded by both a person reference and a reporting verb was four (30.43). Interestingly, this type of enactment was the only one that was in some cases (8.70%) produced without active multimodal articulators.

Table 7. Number of multimodal articulators used in enactments produced by PWA involving different linguistic markers.

	Bare (n= 63)		Only person reference (n= 21)		Only reporting verb (n= 5)		Person reference and reporting verb (n= 23)
Number of active articulators	n	%	n	%	n	%	n	%
Zero	0	0,00%	1	4,76%	0	0,00%	2	8,70%
One	10	15,87%	4	19,05%	0	0,00%	4	17,39%
Two	12	19,05%	3	14,29%	1	20,00%	4	17,39%
Three	21	33,33%	7	33,33%	2	40,00%	6	26,09%
Four	18	28,57%	5	23,81%	2	40,00%	7	30,43%
Five	2	3,17%	1	4,76%	0	0,00%	0	0,00%

In Table 8, the relationship between the presence of linguistic markers and the mean number of multimodal articulators is presented. Enactments preceded by both a person reference and a reporting verb were produced using the least multimodal articulators (M = 2.52), and the enactments preceded by only a reporting verb were produced using the largest number of multimodal articulators (M = 3.20). Bare enactments had a higher mean number of active articulators (2.84) than enactments preceded by (only) a person reference (2.67).

Table 8. Relationship between linguistic markers of enactment and mean number of articulators used by PWA.

Quoting predicate	Number of articulators
	M	SD
Absent	2.84	1.11
Only person reference	2.67	1.32
Only reporting verb	3.20	0.84
Person reference and reporting verb	2.52	1.34

Of the 63 bare enactments (i.e., those that were not preceded by a name, pronoun and/or reporting verb, 48 (76.2%)) were preceded by a shift in intonation. Twenty-six (41.3%) were accompanied by a character viewpoint gesture, and 8 (12.7%) were accompanied by an “other” gesture. The facial expression of the enacted character was mimicked for the realisation of 26 (41.3%) of the bare enactments. Finally, 33 (52.4%) of the bare enactments were accompanied by a posture change (horizontal/vertical/sagittal). The absence of a reporting verb (n = 84, 75% of all enactments) co-occurred with the absence of an intonation shift in only nine cases (10.7%).

PWA vs. NBD communicators

In Table 9 the comparison between the use of multimodal articulators to realise enactment between the participants of the current study and the NBD participants of the study reported by Stec (2016) (n =, p. 26) is presented. Compared to the NBD participants, the PWA use relatively more character intonation, gesture, and character facial expression, and less shifts in gaze and posture.

Table 9. Frequencies and percentages of enactments involving multimodal resources for PWA and NBD communicators (reported by Stec (2016, p. 149)).

	Enactments produced by PWA involving resource		Enactments produced by NBD participants involving resource (Stec, 2016)
Resource	n	%	n	%
Character intonation	87	77.7	389	55.3
Gesture	42	37.5	145	20.6
Character facial expression	58	51.8	336	47.7
Meaningful use of gaze	73	65.3	503	71.4
Posture change	49	43.8	596	84.7

In Table 10, the mean number of multimodal articulators accompanying enactments produced by the PWA is compared to that of enactments produced by Stec (2016)’s NBD participants. The average quantity for the PWA (2.76) is very similar to that for the NBD participants (2.80).

Table 10. Mean number of articulators used to realise enactment by PWA vs. NBD participants (reported by Stec, 2016, p. 150).

Participants	Number of articulators
	M	SD
PWA	2.76	1.187
NBD (Stec, 2016)	2.80	1.205

Multimodality and stance-taking

Finally, the occurrence of affiliation and disaffiliation, and their relationships with shifts in intonation and gesture (Debras, 2015) were assessed. In 58 instances of enactment (51.8%), the PWA affiliated with the enacted character. In 40 instances (35.7%) the PWA disagreed with the enacted character. In four cases (8.9%) the participant did not affiliate nor disaffiliate because the enacted character was fictive, inanimate or an animal. In Tables 11 and 12 the relationships between stance and the use of intonation and gesture in the realisation of enactment by PWA are presented. Enactments representing both disaffiliation and affiliation co-occurred with intonation shifts at the same rate (77.5% and 77.6%, respectively) (Table 11). However, the percentage of co-occurring character viewpoint gestures was higher for enactments representing disaffiliation (45.0%) than those for affiliation (34.5%) (Table 12).

Table 11. Relationship between intonation and stance.

Intonation	Disaffiliation (n= 40)	Affiliation (n= 58)	NA (n= 4)	Unclear (n= 10)
Absent	7.5%	19.0%	0%	0.0%
Present	77.5%	77.6%	75%	80.0%
Unclear	2,5%	1.7%	0%	0.0%
NA	12.5%	1.7%	25%	20.0%

Table 12. Relationship between gesture and stance.

Gesture	Disaffiliation (n= 40)	Affiliation (n= 58)	NA (n= 4)	Unclear (n= 10)
No gesture	42.5%	46.6%	50.0%	80.0%
CVP gesture	45.0%	34.5%	50.0%	20.0%
Other gesture	12.5%	19.0%	0%	0%

Discussion

This study is the first to systematically examine the co-occurrence of a range of important multimodal resources in authentic everyday interactions in PWA, and to recognise the interpersonal, rather than purely referential, role these resources play. While the scope of previous multimodal aphasia research has generally been limited to the use of gesture, pantomime and/or pointing, this study has also explored the concurrent roles of intonation, gaze, facial expression and posture. In addition, rather than seeing these resources as simply a support or even substitute for language, this study examined their roles as independent but interrelated “meaning making” resources.

Linguistic and multimodal characteristics

In terms of our first research question, we found that most enactments were not preceded by a person reference and/or reporting verb (bare enactments). This finding is unsurprising given the nature of aphasia and is in line with previous studies (e.g., Groenewold et al., 2013). It would appear then that the PWA can indeed often successfully convey reported events without explicitly marking these linguistically at all. The fact that PWA used the same multimodal resources (gesture, facial expression, gaze, and body posture) to realise enactments as NBD communicators suggests that these are perhaps the most crucial components or at least important supplementary components and that these modalities can all be considered retained strengths for PWA.

Relationship between linguistic characteristics and number of multimodal articulators

In terms of our second research question – as expected, the mean number of multimodal articulators was lowest for enactments preceded by both a person reference and a reporting verb. Furthermore, the number of multimodal articulators for enactments preceded by only a person reference is lower than that for those preceded by only a reporting verb. This could be due to the PWA needing to indicate who is being enacted in the latter situation, whereas enactments preceded by a person reference require less multimodal marking to “flag” enactment. However, the number of multimodal articulators used by the PWA to realise bare enactments is even lower than that for enactments that are preceded by a reporting verb. This is a somewhat surprising but significant finding that speaks to the issues of modalities other than speech being simply used in a compensatory manner. In other words, there is no clear relationship between the levels of “linguistic bareness” and “multimodal markedness” of enactments in PWA. It could be that reporting verbs are too abstract to simply be compensated for in a non-verbal form and that the abstractness of them may challenge PWA. Degrees of abstraction may be an area for future research in terms of potential meanings involved.

PWA vs NBD communicators

In comparing PWA and NBD communicators, our third research question is answered by the fact that while overall use of multimodal articulators appears to be similar in terms of quantity, PWA clearly utilise the resources differently. Their higher use of intonation, gesture and, to a lesser extent, facial expression (in line with previous findings by Buck and Duffy (1980) and Duffy and Buck (1979)) demonstrates that these are indeed semantic resources for PWA, which could be utilised more in therapeutic endeavours.

PWA also used gaze and posture frequently to realise enactment, but to a lesser extent and in a different way than NBD speakers. For example, whereas NBD speakers direct their gaze away from the listener(s) to indicate that they are entering into a part of a story that will be enacted rather than narrated (Sidnell, 2006), the PWA instead often maintained their gaze with the addressee, or even shifted their gaze towards the addressee. This could be due to the PWA needing to ensure engagement with their communication partner while undertaking the relatively abstract and potentially “difficult” communicative act of enactment. It could also be due to the PWA ensuring that they hold the turn (see, e.g., Laakso, 2014). Finally, it could be due to the PWA needing to check on the partner’s comprehension during this act.

Research in NBD communication on the realisation of enactments has suggested that (shifts in) intonation and gesturing style are important markers for stance-taking: communicators use such shifts to distance themselves from a stance attributed to an absent third party by enactment (Debras, 2015). The findings of the current study indicate a similar pattern for gestures (i.e., relatively more shifts for enactments representing disaffiliation than for enactments representing affiliation), but a different one for intonation (i.e., no difference). This again raises further questions in terms of the use of intonation in meaning-making in PWA and the use of intonation for different functions.

Contribution of this research

Whereas the quantity of the multimodal articulators used by the PWA is similar to that reported for NBD communicators, the qualitative characteristics (i.e., the preference for the type of multimodal articulators, and the interpersonal functions they fulfil) are different. It is the nature of the use of particular patterns of multimodal resources by PWA, and the difference in this usage compared with NBD speakers that are of interest in this study. In addition, showing how PWA “orchestrate” different modalities together to convey evaluative, modalising meanings rather than referential content alone, the current paper contributes to our understanding of interpersonal and stance-taking processes in PWA.

More generally, this research contributes to the literature by raising issues regarding the important notion of multimodality in aphasia research. As argued by Adami (2016) and Jewitt (2014), the examination of relations among modes is key to understanding communication. Applying frameworks that were developed for the examination of multimodal enactment in NBD communication, this study increased our understanding of the co-occurrence of communication modalities such as (shifts in) intonation, gaze, gesture, facial expression, and body posture in PWA. Moreover, unlike most studies assessing multimodality in aphasia, this study relied on authentic interactions, ensuring its ecological validity. Finally, it introduced a useful framework to systematically examine the multimodal characteristics of everyday interaction in aphasia, which could be applied by other researchers.

Limitations

Although this study raised important issues regarding the notion of multimodality and provided innovative insights into the roles multimodal resources can play in interpersonal communication, the findings should be interpreted with caution. Whereas the strength of the study lies in the systematic analysis of naturally occurring interactions between PWA with multiple communication partners in multiple contexts, the findings based on six PWA cannot be generalised. Furthermore, the fact that four of the participants had a (right-sided) hemiparesis may have affected their gesturing and posture movement skills. Whereas such an effect would only reinforce the pattern found for (increased use of) gesture by PWA, it is unclear whether and how it would affect the outcomes with regard to body posture. The relatively small number of enactments produced by each speaker also limits generalisation. While the authentic, spontaneous nature of the data used in this study is one of its strengths, future studies might well consider ways to optimise elicitation of enactments through perhaps guiding topics discussed by participants (e.g., reporting on conversations with others) or simply taking longer samples.

Further research on the multimodal articulators analysed here can reveal to what extent the findings of the current study also apply to different PWA, interactional contexts, and interactional phenomena. Such research would lead to new insights into the interplay of these potentially important “meaning making” resources, informing both aphasia research and intervention.

Acknowledgments

The authors would like to thank the participant volunteers who took part in this research.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work is part of the research programme The use of direct speech as a compensatory device in aphasic interaction with project number [446-16-008], which is financed by the Netherlands Organisation for Scientific Research.