Enhancing Social Messaging with Mediated Social Touch

Abstract

Mediated social touch (MST) is a popular way to communicate emotion and connect people in mobile communication. This article applies MST gestures with vibrotactile stimuli in two online communication modes—asynchronous and synchronous communication (texting and video calling) to enhance social presence for mobile communication. We first designed the application that included the visual design of MST gestures, the vibrotactile stimuli design for MST gestures, and the interface design for texting and video calling. Then, we conducted a user study to explore if the MST gestures with vibrotactile stimuli could increase social presence in texting and video calling compared to MST gestures without vibrotactile stimuli. We also explored if the communication modes affected the social presence significantly when applying MST signals. The quantitative data analysis shows that adding vibrotactile stimuli to MST gestures helps to increase social presence in the aspects of co-presence, perceived behavior interdependence, perceived affective understanding, and perceived emotional interdependence. Adding vibrotactile stimuli to MST gestures causes no significant differences in attentional allocation and perceived message understanding. There is no significant difference between texting and video calling when applying MST signals in mobile communication. The qualitative data analysis shows that participants think MST gestures with vibrotactile stimuli are interesting, and they are willing to use them in mobile communication, but the application design should be iterated based on their feedback.

1. Introduction

Mediated social touch (MST) describes one person touching another person over a distance with tactile or kinesthetic feedback (Haans & Ijsselsteijn, 2006). Some researchers have demonstrated that the mobile device is the most wanted non-wearable device to communicate MST signals (Rognon et al., 2022). The recent haptic technology in mobile devices—embedded vibration actuators, makes it possible to communicate MST signals through mobile devices (Gordon & Zhai, 2019; Rantala et al., 2013). For example, the Taptic Engine has been embedded in iPhones since iPhone 7, which could simulate various vibrotactile effects for different applications (Liu et al., 2018). Some researchers have developed prototypes and applications for mobile devices to communicate MST signals, such as patting, slapping, kissing, tickling, poking, and stroking (Furukawa et al., 2012; Hemmert et al., 2011; Park et al., 2013, 2016; Teyssier et al., 2018; Zhang & Cheok, 2021).

MST signals are important in mobile communication since it can elicit feelings of social presence (van Erp & Toet, 2015). Social presence describes the degree to which a user is perceived as real (Gunawardena & Zittle, 1997; Hadi & Valenzuela, 2020; Oh et al., 2018) and with access to intelligence, intentions, and sensory impressions (Biocca, 2006). Haptic feedback is a popular and useful way to increase social presence and convey more affective information in mediated social interaction (Huisman, 2017; Oh et al., 2018; van Erp & Toet, 2015) during phone calls, video conferencing, and text messaging (van Erp & Toet, 2015). For example, some researchers have applied haptic feedback for mediated social interaction in a collaborative environment (Basdogan et al., 2000; Chellali et al., 2011; Giannopoulos et al., 2008; Sallnäs, 2010; Yarosh et al., 2017) to increase social presence.

However, not too many MST studies focused on mobile devices with haptic feedback in social presence. We will apply the haptic technology (Wei et al., 2022d), using a mobile device embedded with a linear resonance actuator (LRA) to explore the field of MST signals, online communication, and social presence.

The objective of this article is to increase social presence in mobile communication by applying MST gestures with vibrotactile stimuli. In this study, we present the MST gesture design with vibrotactile stimuli for mobile communication. The detailed design factors in this study are the visuals of MST gestures, the vibrotactile stimuli of MST gestures and the interface of mobile communication applications. In the user study, we will explore if the MST gestures with vibrotactile stimuli has a higher social presence than the MST gestures without vibrotactile stimuli. We will also explore if there is a significant difference between synchronous (video calling) and asynchronous communication (texting) in social presence when applying MST signals.

2. Related work

2.1. MST gestures with haptic stimuli on mobile devices

Many researchers developed prototypes for communicating MST gestures with haptic stimuli on mobile devices. Zhang and Cheok (2021) designed a haptic device—Kissenger to display kissing when using mobile devices for online communication to make users feel a deeper emotional connection. The haptic device could capture the real-time sensor data and transmit it to the mobile application and other users over the communication network. Users can use the haptic device for touch communication during video or audio calls. Besides the communication between a dyad, this haptic device can also be used in social applications, such as Skype, Facebook, and WhatsApp for multiple people communicating together. Park et al. (2016) designed CheekTouch to transmit MST signals, such as patting, slapping, tickling, and kissing during a phone call. The haptic prototype was attached to the mobile phone. Users can use different finger gestures on the phone screen to trigger different vibrotactile stimuli on the other party’s cheek. Park et al. (2013) designed POKE to transmit MST signals during a phone call. This prototype was an inflatable surface that could be attached to the back of the mobile device for inputting MST gestures and the front of the mobile device for receiving MST signals. Hoggan et al. (2012) proposed Pressage and ForcePhone. Users can squeeze the side of the phone with different pressures to trigger different vibrations on the recipient’s phone during a phone call. Teyssier et al. (2018) developed MobiLimb, for mobile devices. It could provide MST signals, such as stroking and patting with haptic stimuli. Furukawa et al. (2012) designed KUSUGURI, with this tactile interface, the user could share a body part with another user. The dyads could use this interface for bidirectional tickling. Bales et al. (2011) designed CoupleVIBE, which could be used for person-to-person touch. It sends touch cues between partners’ mobile phones by vibrations to share location information. Pradana et al. (2014) designed a ring-shaped device to transmit MST signals when using mobile phones for telecommunications. Ranasinghe et al. (2020) designed a haptic device—EnPower. Users can customize MST signals on mobile phones and transmit them to a glove for visually impaired people to perceive (Ranasinghe et al., 2020). Zhang et al. (2021) designed SansTouch for remote handshaking with a mobile phone.

Based on the above, we found some space for further investigation:

• Communication modes. Many researchers focused on traditional phone calls, with the phone on the ear. This study will consider other communication modes, such as video calling and texting. Zhang and Cheok (2021) considered video calling, but there was an additional prototype for the mobile device. This study will use a mobile device embedded with an LRA to display vibrotactile stimuli directly without extra devices.

• Touch modality. In the above studies, visual information and haptic information are mostly separated. Wilson and Brewster (2017) has demonstrated that combining multiple modalities could increase the available range of emotional states. This study will apply multimodal MST signals, with visual and haptic information together. We will explore if multi-modal modalities have better feedback than the single modality.

2.2. Social presence with haptic stimuli

Social presence with haptic stimuli has been explored in collaborative environments (Basdogan et al., 2000; Chellali et al., 2011; Giannopoulos et al., 2008; Sallnäs, 2010; Yarosh et al., 2017), remote communication (Nakanishi et al., 2014), affect communication (Ahmed et al., 2020), interaction with a virtual agent (Ahmed et al., 2020; Lee et al., 2017), and behavior influence (Hadi & Valenzuela, 2020). Researchers used haptic devices, such as wearables (Yarosh et al., 2017), robot hands (Chellali et al., 2011; Nakanishi et al., 2014), designed haptic prototypes for hands (Basdogan et al., 2000; Giannopoulos et al., 2008; Lee et al., 2017) and feet (Lee et al., 2017), smartwatches (Hadi & Valenzuela, 2020), and mobile devices (Hadi & Valenzuela, 2020).

Oh et al. (2018) reviewed studies on haptic social presence and found that haptic stimuli influenced social presence significantly. A positive relationship existed between haptic stimuli and perceptions of social presence (Oh et al., 2018).

Based on the above, we observe the following:

• Haptic devices. In social presence, most haptic devices are wearables, robots, and other prototypes developed to display haptic stimuli. Not many studies applied mobile devices in the social presence field. Hadi and Valenzuela (2020) used the mobile device and conducted studies to show that text messages with haptic alerts help improve consumer performance on related tasks. The increased sense of social presence helped to drive this effect. But the context of their work differs from ours. They applied haptic alert messages to influence behavior, while we planned to apply vibrotactile stimuli for MST gestures in mobile communication to increase social presence.

• Contexts. Most studies focused on collaborative tasks [e.g., collaboratively designing and drawing a new logo or poster (Yarosh et al., 2017), and passing an object between two people (Sallnäs, 2010)]. This study will consider a daily casual chatting context without specific collaboration tasks.

2.3. Asynchronous and synchronous MST signals transmission and emotional expressions in mobile application

Synchronous ways include face-to-face and telephone conversations, such as phone calls and video calling (Robinson & Stubberud, 2012). Asynchronous communication is usually conducted through email, online discussion boards, or direct messages, such as texting in WhatsApp, WeChat, or iMessage (Bailey et al., 2021).

Many researchers have developed prototypes and applications for synchronous MST signals in mobile communication. For phone calls, researchers have designed different prototypes, such as Kissenger (Zhang & Cheok, 2021), CheekTouch (Park et al., 2016), POKE (Park et al., 2013), ForcePhone and Pressage (Hoggan et al., 2012), CoupleVIBE (Bales et al., 2011), and Bendi (Park et al., 2015). For video calling, prototypes can communicate MST, such as remote handshaking (Nakanishi et al., 2014) and kissing (Zhang & Cheok, 2021). Other MST signals for synchronous communication, examples are tickling with KUSUGURI (Furukawa et al., 2012), grasping by a mobile sized prototype (Hemmert et al., 2011) and handshaking by SansTouch (Zhang et al., 2021).

Researchers have also developed prototypes and applications for asynchronous communication, adding vibrotactile stimuli to the specific text or using multimodal emojis during texting. Pradana et al. (2014) designed a ring-shaped wearable system—Ring U, which could promote emotional communication between people during texting messages using vibrotactile expressions. MobiLimb (Teyssier et al., 2018) could allow users send a tactile emoji during texting. The other user can feel the tactile emoji on the back of the hand while holding the phone or on the wrist with MobiLimb (Teyssier et al., 2018). Israr et al. (2015) designed Feel Messenger, a social and instant messaging application that provided emojis and expressions with vibrotactile stimuli. Wilson and Brewster (2017) designed Multi-Moji for mobile communication applications, which combined vibrotactile, thermal, and visual stimuli together to expand the affective range of feedback. Wei et al. (2020) proposed a method to design vibrotactile stimuli for emojis and stickers in online chatting applications.

Based on the above, we find that:

• For synchronous communication, most MST signals on mobile devices have a single modality during a phone call, mainly the haptic modality. This article will consider multimodal MST signals in visual and haptic modalities. Besides phone calls, we will consider other communication modes of texting and video calling. Zhang and Cheok (2021) considered kissing on mobile devices during video calls, but the MST types were limited. Users needed to use their lips and cheeks to touch the prototype to feel kissing (Zhang & Cheok, 2021), which might not be convenient for some people. This study will consider designing for a phone on the hands.

• For asynchronous communication, most studies mainly applied multimodal emoji to expand the affective range of feedback. Most emojis express different looks and emotions. There is a lack of considering specific multimodal MST signals on mobile devices. This study will consider multimodal MST signals in online communication.

3. Application design

3.1. Design of mediated social touch

3.1.1. Selection of MST gestures

We explored the physical properties and designed vibrotactile stimuli for 23 MST gestures in our previous studies (Wei et al., 2022a, 2022b). We only choose six MST gestures (“Hit,” “Pat,” “Stroke,” “Nuzzle,” “Push,” and “Hug”) in this study for the following reasons:

• This research is to explore if adding vibrotactile stimuli could increase the social presence and if there is a significant difference between synchronous (video calling) and asynchronous communication (texting) in social presence when applying MST signals. The key point is not the categories of MST gestures. It is not necessary to apply all MST signals.

• Due to the technical limitation at this stage, we cannot apply all MST signals on the interface because the interface is too small to display all MST signals without overlapping.

As touch communicates emotion (Hertenstein et al., 2006, 2009), we want to select MST gestures with different emotional expressions (Hertenstein et al., 2009) and frequently used in mobile communication (Wei et al., 2022a). We selected “Hit,” “Hug,” “Nuzzle,” “Pat,” “Push,” and “Stroke” in this study for the following reasons:

• From the perspective of emotional expression. We found “Hit,” ‘Hug,” “Nuzzle,” “Pat,” “Push,” and “Stroke” could cover rich emotional expressions, such as “Anger,” “Happiness,” “Sadness,” “Disgust,” “Love,” “Gratitude,” and “Sympathy” between male-male, male-female, female-female, female-male (Hertenstein et al., 2009). We do not consider “Fear” in this study, as our dialogue in the user study will not cause fear.

• From the perspective of usage frequency. Wei et al. (2022a) shows that users use “Hug,” “Pat,” and “Stroke” frequently and sometimes use “Hit,” “Nuzzle,” and “Push.” It is effective to apply these MST gestures in this study.

Table 1 shows the possible expressions these selected MST gestures could express (Hertenstein et al., 2009) and the usage frequency in mobile communication (Wei et al., 2022a).

Table 1. Selection of MST gestures.

Selected MST gestures	Emotional expressions^a	Usage frequency^b
“Hit”	Anger	Sometimes used (3.5 < mean ≤ 5)
“Pat”	Happiness, love, gratitude, sympathy	Frequently used (mean ≥ 5)
“Stroke”	Sad, love, sympathy	Frequently used (mean ≥ 5)
“Nuzzle”	Sad	Sometimes used (3.5 < mean ≤ 5)
“Push”	Anger, push	Sometimes used (3.5 < mean ≤ 5)
“Hug”	Happiness, sad, love, gratitude, sympathy	Frequently used (mean ≥ 5)

^aEmotional expressions were extracted from Hertenstein et al. (2009).

^bParticipants filled a 7-point Likert Scale for usage frequency in Wei et al. (2022a). The detailed value of usage frequency of MST signals could be found in Wei et al. (2022a).

3.1.2. Visual design

We need visual hints for MST gesture input. We applied the concrete figure of hands as the sticker for MST gesture input for the following reasons:

• Visual compensation. It is difficult for vibrotactile stimuli to express it as a single modality (Ernst & Banks, 2002; Wei et al., 2022b). To make the receiver understand the accurate MST signals sent by the sender in online communication, we apply the concrete figure of hands to display the accurate MST gestures. To make the sender aware of what MST signals they are sending, we make the sender’s and receiver’s phones display the same concrete figure of hands.

• Demands in the user study. The MST gestures corresponds to the real social touch (RST) gestures in this study. We want the visual information of hands in MST gestures and RST gestures to be similar, to avoid the fantasy of cute figures of hands affecting the research results. We designed the visuals of MST gestures similar to the real hands.

The principles to design visual stimuli of MST gestures are as follows:

• User-defined gestures. We designed visual stickers based on user-defined gestures for MST. The reason is that user-defined gestures are extracted from users’ natural gestures, which can reflect the users’ typical behavior (Wobbrock et al., 2009). We applied the elicitation study (Wobbrock et al., 2009), obtained user-defined gestures for MST, and calculated each user-defined gesture’s agreement rate in our previous study (Wei et al., 2022a).

• Emoji. For some MST gestures that are not easy to understand only based on the figure of user-defined gestures (Wei et al., 2022a), we design it based on Emoji. The reason is that Emoji has been widely used and validated effective to be understood and accepted by users.

Table 2 shows the visual design concepts. We designed the visual sticker based on user-defined gestures for “Hit,” “Pat,” “Stroke,” “Nuzzle,” and “Push.” The visual stickers of these MST gestures are similar to user-defined gestures.

Table 2. Visual design inspiration.

MST gestures	Inspiration	MST gestures	Inspiration
Hit^a		Nuzzle^a
Pat^a		Push^a
Stroke^a		Hug^b

^aThe inspiration for these MST gestures comes from user-defined gestures (Wei et al., 2022a).

^bThe inspiration for ‘Hug’ comes from the current emoji (Smiling Face with Open Hands, 2015).

For “Hug,” Wei et al. (2022a) show that “Hug” and “Squeeze” have the same user-defined gesture with different contexts. Regarding visual modality as the single channel, it is not easy to differentiate the user-defined gestures of these two MST gestures. Wei et al. (2022a) also found that the user-defined gesture for “Hug” was in the metaphor group. So, we refer to Emoji to choose a unique one for “Hug.” Figure 1 shows the visual stickers we designed for the input MST gestures in this study.

Figure 1. The visual design of stickers when inputting MST gestures.

3.1.3. Vibrotactile stimuli

In our previous study, we provided a design process (Wei et al., 2022b) to design vibrotactile stimuli for MST gestures based on the social touch properties, such as pressure and duration (Wei et al., 2022a). We applied the recommended vibrotactile stimuli for these MST gestures (except for “Stoke”). The results showed that the vibrotactile stimuli for “Stroke” were too strong (Wei et al., 2022b). In this study, we changed the frequency of “Stroke” to give it a gentle intensity for an iteration.

Table 3 shows the parameters and recorded accelerations of vibrotactile stimuli for the MST gestures (Wei et al., 2022b).

Table 3. Details of vibrotactile stimuli for the MST gestures selected from Wei et al. (2022b).

Selected MST gestures	Frequency (Hz)	Duration (s)	Recorded accelerations
“Hit”	148	0.15
“Pat”	116	0.10
“Nuzzle”	86–116–86	0.60
“Stroke”	40–70	0.60
“Push”	153	0.60
“Hug”	156	1.00

3.2. Interface design

We developed texting and video calling interfaces with haptic input and display using Android Studio (API: 25).

3.2.1. Interface for texting and sticker animation

Figure 2 shows the interface for texting. The visual stickers of MST gestures are invisible at first. The visual sticker of MST gestures displays when the user touches a specific area. When the user’s finger does not touch the area, the visual sticker of MST gestures becomes invisible again. The positions of the visual stickers are fixed. We set the positions of visual stickers considering two aspects: (1) proper positions in real life and (2) evenly distributed throughout the interface. For example, “Stroke” on the head, “Nuzzle” on the nose, “Hit” in the face, “Pat” on the shoulder, and “Push” the shoulder are all possible gestures in real life. “Hug” around the shoulder could also be possible in real life and for evenly distributed needs in the interface. Figures 2(c,e) show the positions and the display of different MST gestures:

Figure 2. Interface for texting. (a) interface for texting words, (b) interface for the participants’ inputting MST gestures (the avatar is the experimenter), (c) positions of each MST gesture, (d) interface for the participants’ receiving MST gestures (the avatar is the participant), and (e) positions of each MST gesture. We developed this texting interface based on an open-source application (https://github.com/Baloneo/LANC).

• Stroke. We set the position of “Stroke” on the upper head. The visual sticker displays when the user touches the upper head of the photo. The visual sticker of “Stroke” moves laterally along with the user finger’s lateral movement on the touchscreen.

• Nuzzle. We set the “Nuzzle” position on the center of the nose. The visual sticker displays when the user touches the nose. The “Nuzzle” sticker moves back and forth along with the user’s finger’s lateral movement on the touchscreen.

• Hit. We set the position of “Hit” on the left side of the head (Figure 2(c)). The visual sticker displays when the user touches the left side of the head of the photo. There is no lateral movement.

• Pat. We set the position of “Pat” on the right side of the shoulder (Figure 2(c)). The visual sticker displays when the user touches the right shoulder. There is no lateral movement.

• Push. We set the position of “Push” in the middle of two shoulders and at a lower position than “Pat.” The visual sticker displays when the user touches a specific area. There is no lateral movement.

• Hug. We set the position of “Hug” around the middle of two shoulders and at a lower position than “Push.” The visual sticker displays when the user touches a specific area. There is no lateral movement.

The position of each MST gesture on the experimenter’s photo is the same as those on the participant’s photo (Figures 2(c,e)).

3.2.2. Interface for video calls

Figure 3 shows the interface for video calls. The relative positions of stickers in the video calling interface are similar to those in the texting interface. “Stroke” the upper head, “Hit” the left side of the head, “Nuzzle” the center of the nose, “Pat” the right shoulder, “Push” the shoulder, and “Hug” the lower part of the shoulder (Figures 3(c,e)).

Figure 3. Interface for video calls. (a) original interface for video calls, (b) interface for the participants’ inputting MST gestures (the full-screen image is the experimenter), (c) positions of each MST gesture on the experimenter’s image, (d) interface for the participant’s receiving MST gestures (the image box in the upper right corner shows the participant, and (e) positions of each MST gesture on the participant’s image. We developed this video calling interface based on an open-source application (https://github.com/xmtggh/VideoCalling).

For the participant’s image box in the upper right corner of the interface, the relative positions of MST gestures are similar to those on the experimenter’s image (the full-screen image). The size of the stickers is smaller than that of the experimenter’s image (Figures 3(c,e)).

3.3. Interface structure

We developed an application software for texting and video calling. We created the necessary interfaces and functionalities for the user study. We do not consider other interfaces or functionalities irrelevant to this study, such as the login or logout parts.

3.3.1. Texting

Figure 4 shows the structure of texting. There are five layers (Table 4):

Figure 4. Structure of texting.

Table 4. Description of the interface structure of texting.

Layer	Interface	Description
1	Texting interface	Layout of texting interface
2	MST gesture input interface	Layout of MST gestures input interface
3	MST gesture input interface	When inputting MST gestures, the participant’s test phone showed the other person’s image (the experimenter) and the stickers of input MST gestures.
4	MST icons	Pressing the MST icon leads participants to the MST signals display (Layer 5) or MST gestures input (Layer 3) interface.
5	MST signals display interface	When receiving MST signals, the participant’s test phone showed the participants’ image and MST signals the other person (the experimenter) sent to them.

1. Layer 1. This layer shows the texting interface. The basic layout is similar to that in existing applications (e.g., WhatsApp, WeChat, Teams on mobile devices). The primary functions on this page are texting, sending the message, and jumping to Layer 2—the MST gestures input interface. We provided a haptic button () on the left lower corner of this page. On its right are the text input box and the send button.

2. Layer 2 and Layer 3. These two layers show MST gestures input interface. Pressing the haptic button in Layer 1 leads to Layer 2. The central part with the image of the other person is the area for MST gestures input. We provided buttons for confirming () and cancelling () the MST gestures input. When cancelling the MST gestures input, the interface jumps back to Layer 1. When starting to input the MST gestures, Layer 3 shows the sticker of MST gestures. When confirming the MST gestures input, the interface jumps to Layer 4 and automatically sends the textual MST icon (}}◝(ˊᵕˋ)◟{{).

3. Layer 4. Pressing the textual MST icon from the participant’s side (the right side in the interface, Figure 4) leads to Layer 3, which shows what the participant has sent. Pressing the textual MST icon from the other side (the left side in the interface, Figure 4) leads to Layer 5.

4. Layer 5. This layer is the MST signals display interface, which shows what MST signals the other person sends.

3.3.2. Video calling

Figure 5 shows the structure of the interface for video calls. There are three layers (Table 5):

Figure 5. Structure of video calling.

Table 5. Description of the interface structure of video calling.

Layer	Interface	Description
1	Video calling interface	Layout of video calling interface.
2	MST gestures input interface	When inputting MST gestures, the participant’s test phone showed the other person’s synchronous image (the experimenter) and the stickers of input MST gestures.
3	MST signals receive interface	When receiving MST signals, the MST signals would display in the upper right corner, which showed the synchronous image of the participant and the stickers of MST gestures the other person (the experimenter) sent to them.

1. Layer 1. This layer shows the layout of the video calling interface. The basic layout of this interface is similar to that in existing applications (e.g., WeChat or Teams on mobile devices). The primary function on this page is transmitting the synchronous image of the dyads.

2. Layer 2. This layer describes the MST gestures input interface. The display area of the other people is also the MST gestures input area. The participants could input MST gestures directly on the interface when video calling with other people, which seems like the participant is touching other people through the touchscreen face to face.

3. Layer 3. This layer describes the MST signals display interface. The window in the upper right corner displays the synchronous image of the participants and the MST signals that other people send, which seems like the other person is touching the participant through the touchscreen face to face.

4. User study

4.1. Research questions

We design and apply MST signals in online communication. There are two research questions, as follows:

1. If applying the MST gestures with vibrotactile stimuli could increase social presence in texting and video calling compared to MST gestures without vibrotactile stimuli?

2. Is there a significant difference between synchronous (video calling) and asynchronous communication (texting) in social presence when applying MST signals?

4.2. Experiment design

We will test two communication modes—texting and video calling, in two conditions—MST gestures without vibrotactile stimuli and MST gestures with vibrotactile stimuli. We apply a 2 × 2 mixed factorial experimental design. The condition is the within-subjects factor, and the communication mode is the between-subjects factor (Table 6).

Table 6. Descriptions of variables.

	MST gestures with vibrotactile stimuli (visual + haptic feedback)	MST gestures without vibrotactile stimuli (only visual feedback)
Texting	Social presence	Social presence
Video calling	Social presence	Social presence

The independent variables are the condition and the communication mode. The dependent variable is social presence. Visual feedback is included in both conditions.

4.3. Participants

We recruited 40 participants (aged 22–37, mean = 27.13, SD = 3.65, 24 females and 16 males) from the local university to participate in the user study. Participants were asked to wear noise-cancelling headphones to avoid the sounds from the vibrotactile stimuli (Figure 6). All participants have experience using smartphones and online communication applications for texting and video calling.

Figure 6. Test environment. (a) Texting and (b) video calling (the second phone on the desk is used for voice transmission).

Participants were asked to hold the test phone with two hands when texting (Figure 6(a)). When inputting MST gestures, participants were asked to hold the test phone with their left hand and input MST gestures with the right index finger (Figure 6(b)). Although both the thumb and index finger are mainly used for interaction with a smartphone’s touchscreen (Le et al., 2019), we consider using the index finger when inputting MST gestures. The reason is that Table 2 shows that the index finger is more frequently used than the thumb finger in the user-defined gesture for the chosen MST gestures in this study.

4.4. Experiment setup

4.4.1. Apparatus

We used a customized version of the LG V30 smartphone as our test phone, the same as that in our previous study (Wei et al., 2022a, 2022b, 2022d). It contains a Linear Resonance Actuator (LRA) (MPlus 1040). The LRA could convert the drive signals to vibrotactile stimuli.

We have two smartphones (both are LG V30). One is with the software that triggers MST gestures with vibrotactile stimuli (with visual and haptic feedback). The other is with the software triggers MST gestures without vibrotactile stimuli (with only visual feedback). During the user study, if participants need to test MST gestures with vibrotactile stimuli, they will use the one that can provide vibrotactile stimuli. Or they will use the one providing no vibrotactile stimuli to test different conditions.

4.4.2. Experiment environment

Participants sat in front of a desk, wearing noise-cancelling headphones (Airpods Pro), which also had a function of telephony (Figure 6). The experimenter stayed in another room to interact with the participants with texting or video calling (Figure 7). The experiment was a role-play scenario setting. Participants were asked to imagine they were talking with a friend online. The role-play setting could help participants to stay focused on the topic and conversation (Seland, 2006; Simsarian, 2003). The details of the environments are as follows:

Figure 7. Setup of test environment.

• Texting. Participants and the experimenter used the test phones to chat over the local area network (LAN).

• Video calling. Due to the technical limitation, the video call application we developed could not transmit voice well. We provided an iPhone to participants. The headphones were connected to the iPhone. The experimenter used another phone to call the iPhone in the participants’ place. Participants used the test phone for synchronous video images transmission and used the iPhone with headphones for synchronous voices transmission (Figure 7).

4.4.3. Scenario

We try to create a situation in which the participants may feel disappointed then relieved, so that they may use the MST signals to express their emotions. We provide one scenario with two activities: A1—go biking and A2—visiting a garden in the user study. In the scenario, the experimenter and the participant have arranged one activity for the weekends. However, due to some reasons, the arranged activity could not be conducted. The experimenter contacted the participant by texting or video calling to discuss these things and tried to suggest a new activity.

Some participants may prefer A1, while other participants may prefer A2. In order to decrease the preference difference of activity and not let the activity order affect the experimental results, the order of activities provided to each participant is counterbalanced.

When A1—go biking is the arranged activity while A2—visiting a garden is the suggested new activity, the details of A1 and A2 presented to participants are as follows:

• A1: “You (the participant) bought a new bicycle last week, and you want to find a partner to go biking very much. You invited your friend Lucy (the experimenter) to go biking on weekends. Lucy has not used her bike for a long time. However, she forgot to check if her bike was ready to use. On Saturday morning, you were waiting for Lucy to set off. She suddenly found that her bike was broken. She needed to tell you that you could not go biking today (by texting or video calling).”

• A2: “Your friend Lucy (the experimenter) suddenly found that a seasoned garden just opened on this Saturday. This garden was only open for the public twice a year, two weeks in spring, two weeks in autumn. You missed the open time last autumn. You wanted to visit this garden very much. Now is a great chance to go there.”

When A2—visiting a garden is the arranged activity while A1—go biking is the suggested new activity, the details of A2 and A1 presenting to participants are as follows:

• A2: “Your friend Lucy (the experimenter) invited you (the participant) to go to a seasoned garden this weekend. This garden was only open for the public twice a year, two weeks in spring, and two weeks in autumn. You missed the open time last autumn. Now is a great chance to go there. You and Lucy planned to meet at the bus station on Saturday morning. Before leaving the house, Lucy checked the details of the garden. Suddenly she found that she got the opening hour wrong. The garden would open next week in this spring, not this week. So, you could not go to the garden today.”

• A1: “Lucy (the experimenter) found the weather was great today. She wanted to invite you (the participant) to go biking to a forest.”

We gave an example of dialogue in texting (selected from one participant in the user study) to show how the experimenter communicate with the participant, as follows:

Experimenter (initiates the conversation): ‘Sorry, we cannot go to the park today. I remembered a wrong opening hour. It will open next weekend, not this weekend.’

Participant: ‘That is so bad’, ‘}}◝(ˊᵕˋ)◟{{’.

Experimenter (proposes another activity): ‘The weather is great, maybe we could go biking to the forest?’

Participant: ‘Great! Even better!’, ‘}}◝(ˊᵕˋ)◟{{’

Experimenter (suggests a meeting time and location): ‘Then we could meet at your house in 20 minutes. See you.’

Participant: ‘See you.’, ‘}}◝(ˊᵕˋ)◟{{’

The dialogue was a guided discussion. The experimenter needed to guide the discussion direction. We did not limit the exact words and sentences that the participant said. Participants could use MST gestures whenever they wanted to use them. The experimenter and participants worked together to complete the communication.

4.4.4. Measures

We collected quantitative data and qualitative data in the user study.

We applied a measurement of Networked Minds Measure of Social Presence Inventory (NMSPI) (Harms & Biocca, 2004) for quantitative data. This questionnaire has been validated to have the ability to distinguish levels of social presence for mediated interactions (Harms & Biocca, 2004), which is effective for MST signals measurement in this study. There are six dimensions (Table 7). Each dimension has six items in a 7-point Likert scale ranging from one (strongly disagree) to seven (strongly agree). The detailed Likert scale items are in (Harms & Biocca, 2004). An example of the questionnaire is in Table 7. Based on our applications, we introduced what the dimensions and questions mean in Table 7. The whole questionnaire was used to rate the MST signals function in this study.

Table 7. Example questionnaire of NMSPM (Harms & Biocca, 2004).

Dimensions^a	Typical questions^a	Descriptions based on this study^b
Co-presence	I caught my partner’s attention.	Was there any feedback showing that my partner responded to my sending MST signals?
Attentional Allocation	My partner remained focused on me throughout our interaction.	Did my partner notice that I sent MST signals?
Perceived Message Understanding	It was easy to understand my partner.	Could I understand the meaning when my partner sends MST signals to me?
Perceived Affective Understanding	My emotions were not clear to my partner.	Did my partner show that they didn’t quite understand the emotions I want to express through MST signals?
Perceived Emotional Interdependence	I was sometimes influenced by my partner’s moods.	Did the MST signals communicating emotions sent from my partner influence me?
Perceived Behavior Interdependence	My partner’s behavior was closely tied to my behavior.	Could I feel the connection between my partner and me through sending MST signals? Was there any feedback showing that my partner behaved in responding to my MST signals?

^aDimensions and typical questions were from Harms and Biocca (2004).

^bAll the questions should be understood as rating the MST signals function.

For qualitative data, we interviewed participants. For privacy concerns, we only record the voice of participants and only the transcripts were kept and used for analysis. There are two fixed questions and an open discussion for each participant:

• Can you feel that you are touching the other person, or the other person is touching you (MST gestures with vibrotactile stimuli)?

• If this function is available in a real online communication application, will you use it (MST gestures with vibrotactile stimuli)?

• Talking about anything that came to your mind based on the experiences of the test.

4.5. Procedure

We first briefly introduced the test to the participants and provided consent forms and questionnaires.

We introduced the test interface to the participants. As the visual stickers of MST gestures were not visible at first, a training session was needed for participants to get familiar with the test interface and the position of MST gestures.

We have two communication modes—texting and video calling. Twenty participants (P1–P20 in Table 8) test texting, while the other 20 participants (P21–P40 in Table 8) test video calling (between-subjects). We have two conditions—MST gestures with vibrotactile stimuli and MST gestures without vibrotactile stimuli (within-subjects).

Table 8. Test communication modes, conditions, and activities for participants.

	MST gestures with vibrotactile stimuli		MST gestures without vibrotactile stimuli
	Participants	Activity order	Participants	Activity order
Texting	P1–P10	A1 → A2	P1–P10	A2 → A1
	P11–P20	A2 → A1	P11–P20	A1 → A2
Video calling	P21–P30	A1 → A2	P21–P30	A2 → A1
	P31–P40	A2 → A1	P31–P40	A1 → A2

The order of activities providing to each participant is counterbalanced. Table 8 shows the activities and the order for participants.

After the introduction and the training session, the detailed procedure for each participant was as follows:

1. Participants first tried one communication mode with one condition, having online communication with the experimenter. For example, a participant first tried using MST gestures without vibrotactile stimuli when texting.

2. Participants filled the 7-point Likert scale on the questionnaire for subjective ratings. (e.g., subjective ratings for MST gestures without vibrotactile stimuli when texting).

3. Participants tried the same communication mode with the other condition, having online communication with the experimenter. For example, the participant tried using MST gestures with vibrotactile stimuli when texting.

4. Participants filled the 7-point Likert scale on another questionnaire for subjective ratings (e.g., subjective ratings for MST gestures with vibrotactile stimuli when texting).

5. The experimenter interviewed the participants.

5. Results

5.1. Quantitative results

5.1.1. Between-subjects analysis

We used SPSS 23.0 to analyze data. We conducted the Shapiro–Wilk test for the normality test because Mishra et al. (2019) showed that this method was appropriate for small sample sizes analysis (<50 samples). All data sets in each dimension conformed to normalized distribution (p > 0.01 in behavior interdependence, p > 0.05 in the rest five dimensions). There were no interactive effects between the communication mode and the condition in all dimensions (p > 0.05).

We conducted one-way MANOVA to test if the communication mode matters in each dimension of social presence. Figure 8 showed no significant differences between different communication modes in each dimension of social presence (p > 0.05). This indicated there was no significant difference between texting and video calling when applying the MST signals in online communication. We also found no significant gender effects during the evaluation (p > 0.05).

Figure 8. Dimensions of social presence. Between-subjective analysis—video calling and texting (VT: vibrotactile stimuli).

5.1.2. Within-subjects analysis

We conducted paired sample t-test to explore if adding vibrotactile stimuli to MST gestures matters when the difference value follows the normalized distribution (Liang et al., 2019), or we conducted the Wilcoxon rank test (Meek et al., 2007). All data sets conformed to normalized distribution (p > 0.05) except for perceived messaging understanding in video calling (p = 0.001).

Figure 9 showed that MST gestures with vibrotactile stimuli provided significantly higher co-presence, perceived behavior interdependence, perceived affective understanding, perceived emotional interdependence than MST gestures without vibrotactile stimuli (p < 0.05). No significant differences could be found in perceived message understanding and attentional allocation of social presence (p > 0.05).

Figure 9. Dimensions of social presence. Within-subjects analysis—with VT and without VT (VT: vibrotactile stimuli).

5.2. Qualitative results

5.2.1. Thematic analysis

We conducted a thematic analysis (Braun & Clarke, 2012) for qualitative analysis. We recorded the voices of participants during the interview and the recordings were transcribed as qualitative data. After familiarizing with the transcripts of participants’ ideas (phase 1 in Braun & Clarke, 2012), we provided initial codes (descriptive sentences in Tables 9 and 10) based on participants’ interviews (phase 2 in Braun & Clarke, 2012). We inducted themes (phase 3 to phase 5 in Braun & Clarke, 2012). We will provide iterative recommendations based on the quantitative analysis (phase 6 in Braun & Clarke, 2012) in the later Discussion section.

Table 9. Qualitative analysis of texting.

Possibility to touch or be touched?	I could feel that I was touching or being touched by other people (8). I could feel that I was touching the other person, but I could not feel the other person touching me (6). I could not feel that I was touching or being touched by other people (6).
Use this function or not?	I will use this function (9). I will use this function with some conditions (8). I will not use this function (3).
Open questions	Touch	Type	Too few types (4).
		Position	Positions of MST gestures were too close and not flexible, the positions could not be remembered (3); Gesture recognition could be considered (1).
		Visual sticker	The meaning of the stickers could not be remembered (1); The stickers were simple (1); Gif animation or special effects may be better (1).
		Vibrotactile stimuli	No big differences among these vibrotactile stimuli (9); The meaning of vibrotactile stimuli could no be remembered (2); The likelihood to be understood as a specific touch for some vibrotactile stimuli was low (2).
	Interactive interface		It was not convenient to jump to another page when using this function (7).
	Avatar		It was strange to input MST gestures on the photo of a person (5); It was interesting to input MST gestures on the photo of a person (2); A dynamic virtual avatar could be considered (1).
	Multimodal MST signals		Visual information was dominant and vibrotactile information is auxiliary (2); Vibrotactile stimuli helped to differentiate indistinguishable visual stickers (1).
	Customized demands		Provide users with a chance to choose whether to apply the vibrotactile stimuli or not. Customizing the avatar size and position and MST types and positions could be considered. Page preview could be considered (3).
	Other factors		Context (1); Familiarity with chatting partner (3); Culture (1).

Table 10. Qualitative analysis of video calling.

Possibility to touch or be touched?	I could feel that I was touching or being touched by other people (9). I could feel I was touching other people to some extent, not full extent (1). I could feel that I was touching the other person, but I could not feel the other person touching me (2). I could not feel that I was touching or being touched by other people (8).
Use this function or not?	I will use this function (18). I will use this function with some conditions (2).
Open questions	Touch	Type	Too few types (3).
		Position	Positions of MST gestures were too close and not flexible, the position could not be remembered (5); Gesture recognition could be considered (2).
		Visual sticker	Gif animations or special effects may be better (3).
		Vibrotactile stimuli	It was easy to attract other people's attention (2); There were no big differences among these vibrotactile stimuli (6); It was difficult to differentiate the touch between the participant and the experimenter (4); The likelihood to be understood as a specific touch for some vibrotactile stimuli was low (2); Vibrotactile stimuli increased the interactive feelings (1); Participants were not used to the right hand performing gestures while the left felt touch (2).
	Interface layout		Focus on the full screen, ignore the small box in the upper right corner (3); Notice the small box in the upper right corner with vibrotactile stimuli, or cannot notice it (1).
	Emotional expression		It was easy to express emotion (4); Participants felt more emotional expressions in sending than they could receive (1).
	Multimodal MST signals		Visual information was dominant and vibrotactile information was auxiliary (1); Audio feedback could be considered (1).
	Customized demands		Social touch type and position, sticker preview (3).
	Other factors		Culture (1).

5.2.2. Texting

For the first question about if participants could touch or be touched by other people via MST signals when texting, participants mainly provided three types of answers (Table 9). Eight participants said they could touch or be touched via this function. Six participants mentioned they could touch others by MST signals, but it was difficult to be touched via this function. P6 and P9 said that they felt they could touch others via MST signals. When receiving the MST signals, they just felt vibrations on their hands, or the mobile phone was vibrating. This suggests that gestures with vibrotactile stimuli and visual information might make the touch more real. Six participants could not touch or be touched via this function. Two of them said this function provided a way to express emotion rather than send touch. One participant could feel the vibration when interacting with the touchscreen, but that vibration could not be regarded as social touch. The rest participants said they were just interacting with the touchscreen or with an electronic pet, or just inputting some gestures on a photo, which could not make them touch or be touched by others via MST signals.

For the second question about if participants would use this function in online communication, three types of answers were provided (Table 9). Nine participants mentioned that they would use this function in online communication. Eight participants said that they would use this function with conditions. Participants mentioned that they would be attracted more by a more straightforward and more rational interactive interface, and more types of interesting MST signals. And they also wanted to customize virtual and cartoon avatars. One participant mentioned he/she would use this function with familiar people. For unfamiliar people, texting with words was enough. Two participants said they got used to using only words rather than MST signals, emojis, or stickers during texting.

For answers to open questions, we found six themes, including Touch, Interactive interface, Avatar, Multimodal MST signals, Customized demands, and other factors (Table 9). The Touch theme has four dimensions, including Type, Position, Visual sticker, and Vibrotactile stimuli (Table 9). Table 9 shows the themes and descriptions.

5.2.3. Video calling

For the first question about if participants could touch or be touched by other people via MST signals during video calls, participants mainly provided four types of answers (Table 10). Nine participants said they could touch or be touched by this function. Two participants mentioned they could touch other people via MST signals, but it was difficult to be touched via MST signals. P9 said she could “Hug” and “Stroke,” but the “Hit” was too gentle. Eight participants could not touch or be touched via this function. Four of them said they could feel vibrations, which could not be regarded as social touch. Two of them mentioned they could feel strong emotional expressions rather than social touch. Two participants mentioned the right hand was performing gestures, but the left hand felt vibrations, which made it difficult to touch or be touched via this function.

For the second question about if participants would use this function in online communication, two types of answers were provided (Table 10). Eighteen participants would use this function in online communication. Two participants would use this function with conditions. One of them mentioned he/she would use this function with familiar people. The other one said he would use it if the application could be designed better.

For answers to open questions, we found six themes, including Touch, Interface layout, Emotional expression, Multimodal MST signals, Customized demands, and other factors (Table 10). The Touch theme has four dimensions, including Type, Position, Visual sticker, and Vibrotactile stimuli (Table 10). Table 10 shows the themes and descriptions.

6. Discussions

6.1. MST gestures with vibrotactile stimuli and social presence

The research results showed that MST gestures with vibrotactile stimuli increased the social presence in general. This result conforms to previous studies about social presence, which indicates that haptic stimuli help to increase social presence in remote communication in different contexts (Ahmed et al., 2020; Basdogan et al., 2000; Chellali et al., 2011; Giannopoulos et al., 2008; Hadi & Valenzuela, 2020; Lee et al., 2017; Nakanishi et al., 2014; Oh et al., 2018; Sallnäs, 2010; Yarosh et al., 2017).

In this study, we further analyzed different dimensions of social presence. Adding vibrotactile stimuli to MST gestures helped increase social presence in co-presence, perceived behavior interdependence, perceived affective understanding, and perceived emotional interdependence. Adding vibrotactile stimuli to MST gestures caused no significant differences in social presence from attentional allocation and perceived message understanding. We discuss each dimension as follows:

1. Attentional allocation means the amount of attention the user allocates to and receives from an interactant (Harms & Biocca, 2004). There were no significant differences in attentional allocation, no matter if there were vibrotactile stimuli or not, in both video calling and texting. Possible reasons are as follows:

   • For video calling, participants’ attention was mostly on inputting MST gestures, which might lead to no significant differences in attentional allocation. Many participants mentioned that they paid more attention to their own manipulations. They did not pay much attention to MST signals sent from the other person. Maybe the size of the right upper box showing their own images (Figure 3) was too small to be noticed no matter if there were vibrations or not.

   • For video calling, the hard-to-differentiate vibrotactile stimuli might cause no significant differences in attentional allocation. Many participants mentioned the vibrotactile stimuli for each MST gesture were difficult to differentiate. Sometimes, the dyads were sending MST signals simultaneously. They just felt the mobile device was vibrating, but they did not know who triggered these vibrations. The vibrotactile stimuli could not work well in this dimension. Users’ confusions might result in similar attentional allocation no matter if there were vibrations or not.

   • For texting, the interaction—“going to the next page” may cause no big differences in attentional allocation. The haptic icon led participants to the next page for inputting and perceiving the MST signals. Many participants mentioned that it was inconvenient to press a button and go to the next page for further interaction. Participants wanted to input or receive MST signals just on the texting page directly. If participants went to another page, the process—“going to the next page” might get their attention rather than the vibrotactile stimuli.

   • For texting, the interaction—“poking a photo” may affect participants’ attentional allocation. Many participants mentioned that “poking a photo” was strange, which might have caught too much of attention. This situation might cause no significant differences in this dimension during texting.

2. For perceived message understanding, MST signals are also presented with stickers in this study. No matter if there were vibrotactile stimuli or not, participants could easily know what the other person wanted to express.

3. For co-presence, many participants mentioned MST gestures with vibrotactile stimuli was interesting. Although many participants said they could not differentiate the vibrotactile stimuli during video calling since sometimes the vibrotactile stimuli triggered by the participant and the experimenter displayed simultaneously, the vibrotactile stimuli could still make them feel something.

4. For perceived affective understanding and perceived affective interdependence, the results showed that MST gestures with vibrotactile stimuli provided significantly higher social presence than MST gestures without vibrotactile stimuli in these two dimensions. This result conforms to other related studies, which have demonstrated that vibrotactile stimuli help express emotions better (Akshita et al., 2015; Pradana et al., 2014; Takahashi et al., 2017; Wilson & Brewster, 2017).

5. For perceived behavioral interdependence, many participants mentioned that they would like to use MST signals with vibrotactile stimuli to express their emotions, which could be a new way for them to express emotion.

Based on the above, we found that vibrotactile stimuli mainly help increase social presence in aspects of co-presence, perceived behavior interdependence, perceived affective understanding, and perceived emotional interdependence. And it would not affect the message understanding and attentional allocation in mobile communication.

6.2. Communication mode and social presence

Existing studies mainly focused on one communication mode [e.g., phone calls (Hoggan et al., 2012; Park et al., 2013, 2015, 2016), texting (Pradana et al., 2014), or video calling (Nakanishi et al., 2014)]. This study applied two communication modes together and compared whether MST caused differences in these two communication modes.

The result showed no significant differences between texting and video calling when applying MST signals in online communication. Although there might be limitations in the current design, interview results showed that most participants found the interaction was interesting. Participants were willing to use it.

For future design, we need to have a deeper insight into users’ needs of MST signals in different communication modes. We need to apply the MST signals to different modes based on the users’ possible different needs in different modes.

6.3. Implications for future design

We discuss implications for future design based on participants’ interview results, from the perspective of vibrotactile stimuli, visual design, interface design, and interface structure.

6.3.1. Vibrotactile stimuli for MST gestures

About the vibrotactile stimuli for MST gestures, we provided some implications for future design as follows:

• More compound waveform composition types (Wei et al., 2022b, 2022d) are needed. Many participants felt the vibrotactile stimuli were similar. They could only feel the length difference (long and short). These comments on vibrotactile stimuli illustrated that participants were willing to feel more interesting and changing vibrotactile stimuli than simple short pulse or long and unchanging vibrotactile stimuli.

• Applying multi-modal stimuli to increase the perceiving of MST signals is necessary. We should notice that people may not have the same perceiving of one MST signal in different contexts or different communication modes. This result conforms to the previous study (Salminen et al., 2009), which shows the vibrotactile stimuli could be more pleasant and less arousing in the bus than in the laboratory. We suggest increasing the perceiving of MST signals in other ways, such as using multi-modal stimuli (haptic, visual, audio, thermal, et al.) because Wilson and Brewster (2017) have demonstrated that multi-modal stimuli increase the available range of perception. For example, some researchers developed Multi-moji (Wilson & Brewster, 2017), which combined vibrotactile, visual, and thermal stimuli together, and VibEmoji (An et al., 2022), which provided vibrotactile stimuli, animation effects, and emotion stickers together in mobile communication.

• Making the vibrotactile stimuli and visual stickers more matched in a specific context and communication mode could be considered. We added visual information and context in this study. Ernst and Banks (2002) mentioned that vision and touch both provided information for estimating the MST signals. Visual information is helpful when judging size, shape, or position (Ernst & Banks, 2002). The dominant channel to feel the MST signals may be different between the only vibrotactile stimuli condition (Wei et al., 2022b) and the vibrotactile stimuli along with visual and context information in this study. The vibrotactile stimuli were designed based on metaphorical cues, while the visual stickers were designed based on gesture movements. It may have caused confusion. The possible solutions are iterating the vibrotactile stimuli or the visual stickers to make them more matched in a specific context and communication mode. For example, in this study, the vibrotactile stimulus for “Hug” was designed to express a feeling of force exerted on other people (Wei et al., 2022b). But the visual sticker of “Hug” comes from emoji, which may not express the feeling of force. The vibrotactile stimuli and visual sticker may not match. We may add varying colors or a progress bar to express a metaphorical force change rather than a simple touch sticker.

• We consider designing and iterating vibrotactile stimuli in a specific context with a specific communication mode. Some participants said they focused more on sending MST signals. The sense of touch was stronger when sending MST signals. When sending MST signals, participants performed gestures and actively felt the vibrotactile stimuli. When receiving MST signals, participants passively felt the vibrotactile stimuli without gestures. However, we tested vibrotactile stimuli from these two situations—sending with gestures and receiving without gestures (Wei et al., 2022c). There were no significant differences between these two conditions on the likelihood to be understood as a specific MST gesture. These contradictory situations show that the context plays an important role in MST communication. In our earlier study (Wei et al., 2022c), there was no context (no video calling or texting). Participants were asked to press the graphic button on the touchscreen, triggering vibrotactile stimuli. Participants could focus more on the vibrotactile stimuli. In this study, more contexts and visual information required attention and participants could not focus as much on the vibrotactile stimuli. So, iterating and testing vibrotactile stimuli in a specific context with a specific communication mode may inspire the design.

6.3.2. Visual design of MST gestures

About the visual design of MST gestures, we suggested as follows:

• Richer visual design is needed. In the user study, we applied simple hand gesture stickers in the application. However, many participants said that the visual design of MST gesture could be more interesting. For example, 3D effects could be considered.

• Customization is needed. Some participants suggested cuter stickers. “Cute” is a concept that can be culture related, and it may suggest that the cultural background should be taken into consideration. Customized MST types and visual stickers could contribute to this point. For example, Memoji and Digital Touch in iMessage could make users customize their stickers and touch gestures.

6.3.3. Interface design

About the interface design, the layout iteration, preview function, and possibility for customization are design opportunities, as follows:

• The layout of the video calling interface needs iteration. The box showing the participants’ image and received MST signals in the right upper corner (Figure 3) was too small to notice. Future interface design could consider enlarging the small box to solve this problem. We could make the size of two split screens equal.

• Pre-view MST gestures before sending is needed. Many participants said they could not remember the MST gestures’ position on the interface as all MST gestures were invisible at first. We could present participants pre-view window for MST gestures.

• Participants also wanted to customize the MST gestures’ types and positions. We only set six typical MST gestures and provided fixed positions for each MST gesture in this study. Some participants said the present MST signals could not express some of their feelings and they hoped the positions could be flexible. We designed 23 MST gestures and provided at least one CWC type of vibrotactile stimuli for each MST gesture in (Wei et al., 2022b). We could consider adding the MST gestures list and vibrotactile stimuli list and making participants choose their preferred and frequently used MST gestures with vibrotactile stimuli.

6.3.4. Interface structure

Interface structure needs iteration as follows:

• In texting, we need to integrate the MST gestures inputting and perceiving function in the texting interface rather than creating a separate interface. Many participants mentioned that “going to the next page” was not convenient. However, the existing texting interface already has a clear layout for users to text. We need to consider how to integrate the MST signals functions in this layout.

• There should be a time difference between the vibrotactile stimuli triggered by the two people in the communication during video calling. Participants said it was difficult to differentiate whether the vibrotactile stimuli were from themselves or the experimenter. Consider providing a time difference between the vibrotactile stimuli triggered by the two people in the communication so that users can perceive them more clearly.

6.4. Limitations

Some participants mentioned that they could feel the vibrotactile stimuli on their left hand (holding the mobile device). Their right hand performed gestures but could not feel the vibrotactile stimuli. For vibrotactile stimuli, it is difficult to solve this problem only on a smartphone. Maybe wearables could solve this problem, but use of the wearables means additional load for the users. Another possible way to solve this problem is to consider audio together with visual and vibrotactile stimuli. The richer modalities may help make people not focus on one modality, making the MST signals more realistic. Our test device has an LRA, which could provide vibrotactile and audio stimuli together. Based on our system, we have already investigated how to design vibrotactile and audio stimuli together (Wei et al., 2022d). We can consider combining the audio stimuli in future designs.

Our user study was based on the selected six MST gestures, which could be frequently used and covered rich emotional expressions. The results are effective based on these six MST gestures. We still have 17 more MST gestures. In future research, we will consider providing more choices and making users experience more MST gestures.

7. Conclusions and future work

This study applied MST gestures with vibrotactile stimuli in mobile communication (texting and video calling). We conducted a user study and found no significant difference between texting and video calling when applying MST signals in online communication. Adding vibrotactile stimuli to MST gestures helps to increase social presence in the aspects of co-presence, perceived behavior interdependence, perceived affective understanding, and perceived emotional interdependence. Adding vibrotactile stimuli to MST gestures causes no significant differences in attentional allocation and perceived message understanding. Participants thought MST gestures with vibrotactile stimuli was attractive and they were willing to use it, but the application design should be improved based on users’ needs.

For future work, we need to improve the design of vibrotactile stimuli for some MST gestures, visual designs of the MST gestures, and interface design for different communication modes. We can also consider vibrotactile, visual and audio stimuli together to create a richer MST signal effect. We will also consider providing more MST signals and making users experience more MST signals in mobile communication.

Ethical approval

This study was conducted according to the guidelines of the local Ethical Review Board from the Department of Industrial Design (protocol code: ERB2021ID15).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Qianhui Wei

Qianhui Wei is a PhD candidate in the Department of Industrial Design, Eindhoven University of Technology, with research topics including social haptics, multimodal user interface, and human–computer interaction.

Jun Hu

Jun Hu is an associate professor of design research on social computing with the Department of Industrial Design, Eindhoven University of Technology.

Min Li

Min Li is a parttime associate professor with the Eindhoven University of Technology. He is a Member of the IEEE Design and Implementation of Signal Processing Systems Technical Committee (IEEE-DISPS TC).