Enhancing Social Messaging with Mediated Social Touch

Figures & data

Table 1. Selection of MST gestures.

Selected MST gestures	Emotional expressions^a	Usage frequency^b
“Hit”	Anger	Sometimes used (3.5 < mean ≤ 5)
“Pat”	Happiness, love, gratitude, sympathy	Frequently used (mean ≥ 5)
“Stroke”	Sad, love, sympathy	Frequently used (mean ≥ 5)
“Nuzzle”	Sad	Sometimes used (3.5 < mean ≤ 5)
“Push”	Anger, push	Sometimes used (3.5 < mean ≤ 5)
“Hug”	Happiness, sad, love, gratitude, sympathy	Frequently used (mean ≥ 5)

^aEmotional expressions were extracted from Hertenstein et al. (2009).

^bParticipants filled a 7-point Likert Scale for usage frequency in Wei et al. (2022a). The detailed value of usage frequency of MST signals could be found in Wei et al. (2022a).

Table 2. Visual design inspiration.

MST gestures	Inspiration	MST gestures	Inspiration
Hit^a		Nuzzle^a
Pat^a		Push^a
Stroke^a		Hug^b

^aThe inspiration for these MST gestures comes from user-defined gestures (Wei et al., 2022a).

^bThe inspiration for ‘Hug’ comes from the current emoji (Smiling Face with Open Hands, 2015).

Figure 1. The visual design of stickers when inputting MST gestures.

Table 3. Details of vibrotactile stimuli for the MST gestures selected from Wei et al. (2022b).

Selected MST gestures	Frequency (Hz)	Duration (s)	Recorded accelerations
“Hit”	148	0.15
“Pat”	116	0.10
“Nuzzle”	86–116–86	0.60
“Stroke”	40–70	0.60
“Push”	153	0.60
“Hug”	156	1.00

Figure 2. Interface for texting. (a) interface for texting words, (b) interface for the participants’ inputting MST gestures (the avatar is the experimenter), (c) positions of each MST gesture, (d) interface for the participants’ receiving MST gestures (the avatar is the participant), and (e) positions of each MST gesture. We developed this texting interface based on an open-source application (https://github.com/Baloneo/LANC).

Figure 3. Interface for video calls. (a) original interface for video calls, (b) interface for the participants’ inputting MST gestures (the full-screen image is the experimenter), (c) positions of each MST gesture on the experimenter’s image, (d) interface for the participant’s receiving MST gestures (the image box in the upper right corner shows the participant, and (e) positions of each MST gesture on the participant’s image. We developed this video calling interface based on an open-source application (https://github.com/xmtggh/VideoCalling).

Figure 4. Structure of texting.

Table 4. Description of the interface structure of texting.

Layer	Interface	Description
1	Texting interface	Layout of texting interface
2	MST gesture input interface	Layout of MST gestures input interface
3	MST gesture input interface	When inputting MST gestures, the participant’s test phone showed the other person’s image (the experimenter) and the stickers of input MST gestures.
4	MST icons	Pressing the MST icon leads participants to the MST signals display (Layer 5) or MST gestures input (Layer 3) interface.
5	MST signals display interface	When receiving MST signals, the participant’s test phone showed the participants’ image and MST signals the other person (the experimenter) sent to them.

Figure 5. Structure of video calling.

Table 5. Description of the interface structure of video calling.

Layer	Interface	Description
1	Video calling interface	Layout of video calling interface.
2	MST gestures input interface	When inputting MST gestures, the participant’s test phone showed the other person’s synchronous image (the experimenter) and the stickers of input MST gestures.
3	MST signals receive interface	When receiving MST signals, the MST signals would display in the upper right corner, which showed the synchronous image of the participant and the stickers of MST gestures the other person (the experimenter) sent to them.

Table 6. Descriptions of variables.

	MST gestures with vibrotactile stimuli (visual + haptic feedback)	MST gestures without vibrotactile stimuli (only visual feedback)
Texting	Social presence	Social presence
Video calling	Social presence	Social presence

Figure 6. Test environment. (a) Texting and (b) video calling (the second phone on the desk is used for voice transmission).

Figure 7. Setup of test environment.

Table 7. Example questionnaire of NMSPM (Harms & Biocca, 2004).

Dimensions^a	Typical questions^a	Descriptions based on this study^b
Co-presence	I caught my partner’s attention.	Was there any feedback showing that my partner responded to my sending MST signals?
Attentional Allocation	My partner remained focused on me throughout our interaction.	Did my partner notice that I sent MST signals?
Perceived Message Understanding	It was easy to understand my partner.	Could I understand the meaning when my partner sends MST signals to me?
Perceived Affective Understanding	My emotions were not clear to my partner.	Did my partner show that they didn’t quite understand the emotions I want to express through MST signals?
Perceived Emotional Interdependence	I was sometimes influenced by my partner’s moods.	Did the MST signals communicating emotions sent from my partner influence me?
Perceived Behavior Interdependence	My partner’s behavior was closely tied to my behavior.	Could I feel the connection between my partner and me through sending MST signals? Was there any feedback showing that my partner behaved in responding to my MST signals?

^aDimensions and typical questions were from Harms and Biocca (2004).

^bAll the questions should be understood as rating the MST signals function.

Table 8. Test communication modes, conditions, and activities for participants.

	MST gestures with vibrotactile stimuli		MST gestures without vibrotactile stimuli
	Participants	Activity order	Participants	Activity order
Texting	P1–P10	A1 → A2	P1–P10	A2 → A1
	P11–P20	A2 → A1	P11–P20	A1 → A2
Video calling	P21–P30	A1 → A2	P21–P30	A2 → A1
	P31–P40	A2 → A1	P31–P40	A1 → A2

Figure 8. Dimensions of social presence. Between-subjective analysis—video calling and texting (VT: vibrotactile stimuli).

Figure 9. Dimensions of social presence. Within-subjects analysis—with VT and without VT (VT: vibrotactile stimuli).

Table 9. Qualitative analysis of texting.

Possibility to touch or be touched?	I could feel that I was touching or being touched by other people (8). I could feel that I was touching the other person, but I could not feel the other person touching me (6). I could not feel that I was touching or being touched by other people (6).
Use this function or not?	I will use this function (9). I will use this function with some conditions (8). I will not use this function (3).
Open questions	Touch	Type	Too few types (4).
		Position	Positions of MST gestures were too close and not flexible, the positions could not be remembered (3); Gesture recognition could be considered (1).
		Visual sticker	The meaning of the stickers could not be remembered (1); The stickers were simple (1); Gif animation or special effects may be better (1).
		Vibrotactile stimuli	No big differences among these vibrotactile stimuli (9); The meaning of vibrotactile stimuli could no be remembered (2); The likelihood to be understood as a specific touch for some vibrotactile stimuli was low (2).
	Interactive interface		It was not convenient to jump to another page when using this function (7).
	Avatar		It was strange to input MST gestures on the photo of a person (5); It was interesting to input MST gestures on the photo of a person (2); A dynamic virtual avatar could be considered (1).
	Multimodal MST signals		Visual information was dominant and vibrotactile information is auxiliary (2); Vibrotactile stimuli helped to differentiate indistinguishable visual stickers (1).
	Customized demands		Provide users with a chance to choose whether to apply the vibrotactile stimuli or not. Customizing the avatar size and position and MST types and positions could be considered. Page preview could be considered (3).
	Other factors		Context (1); Familiarity with chatting partner (3); Culture (1).

Table 10. Qualitative analysis of video calling.

Possibility to touch or be touched?	I could feel that I was touching or being touched by other people (9). I could feel I was touching other people to some extent, not full extent (1). I could feel that I was touching the other person, but I could not feel the other person touching me (2). I could not feel that I was touching or being touched by other people (8).
Use this function or not?	I will use this function (18). I will use this function with some conditions (2).
Open questions	Touch	Type	Too few types (3).
		Position	Positions of MST gestures were too close and not flexible, the position could not be remembered (5); Gesture recognition could be considered (2).
		Visual sticker	Gif animations or special effects may be better (3).
		Vibrotactile stimuli	It was easy to attract other people's attention (2); There were no big differences among these vibrotactile stimuli (6); It was difficult to differentiate the touch between the participant and the experimenter (4); The likelihood to be understood as a specific touch for some vibrotactile stimuli was low (2); Vibrotactile stimuli increased the interactive feelings (1); Participants were not used to the right hand performing gestures while the left felt touch (2).
	Interface layout		Focus on the full screen, ignore the small box in the upper right corner (3); Notice the small box in the upper right corner with vibrotactile stimuli, or cannot notice it (1).
	Emotional expression		It was easy to express emotion (4); Participants felt more emotional expressions in sending than they could receive (1).
	Multimodal MST signals		Visual information was dominant and vibrotactile information was auxiliary (1); Audio feedback could be considered (1).
	Customized demands		Social touch type and position, sticker preview (3).
	Other factors		Culture (1).

Hertenstein, M. J., Holmes, R., McCullough, M., & Keltner, D. (2009). The communication of emotion via touch. Emotion, 9(4), 566–573. https://doi.org/10.1037/a0016108

 PubMed Web of Science ®Google Scholar

Wei, Q., Hu, J., & Li, M. (2022a). User-defined gestures for mediated social touch on touchscreens. Personal and Ubiquitous Computing, 1–16. https://doi.org/10.1007/s00779-021-01663-9

 Google Scholar

Smiling Face with Open Hands (2015). [WWW Document]. Emojipedia. https://emojipedia.org/hugging-face/

 Google Scholar

Wei, Q., Hu, J., & Li, M. (2022b). Designing mediated social touch signals.

 Google Scholar

Harms, C., & Biocca, F. (2004). Internal consistency and reliability of the networked minds measure of social presence. In Seventh Annual International Workshop: Presence, 2004 (pp. 246–251).

 Google Scholar