How Interaction Mechanism and Error Responses Influence Users’ Responses to Customer Service Chatbots

Abstract

As more and more people regularly interact with chatbots in their everyday lives, it is crucial to understand how users perceive and evaluate them. This study examined whether free text (vs. button) interaction with and social (vs. neutral) error responses of chatbots effectively improve user responses. Using an online experiment (N = 416) in which participants interacted with a customer service chatbot, we investigated how interaction mechanism and error response influence perceived warmth and competence, satisfaction, usage intention, and emotional connection with the company. Contrary to our hypotheses, we found no evidence that free text interaction increases acceptance. Interacting via buttons was preferred, potentially because participants perceived the free text chatbot as less warm and competent. Interestingly, social error responses only increased satisfaction. We discuss the results considering recent advances in Large Language Models and highlight how the findings can guide further research and help practitioners develop efficient, user-friendly chatbots.

Keywords:

• Chatbot
• interaction mechanism
• error response
• customer service
• satisfaction
• intention to use
• large language model

Even before ChatGPT (OpenAI, 2023) and Large Language Models (LLM) started to dominate the headlines around the globe in 2023, improvements in artificial intelligence (AI) and automation drove the proliferation of chatbots in various sectors and will continue to do so in the future. Implementing chatbot technology enables companies to save staff and training costs by automating customer service processes like recommending products, managing complaints, and answering questions (Desai, 2023; Rivas & Zhao, 2023). According to a recent study by Gartner, chatbots will become the primary customer service channel for about a quarter of companies by 2027 (Costello & LoDolce, 2022). Accordingly, more and more people will regularly interact with chatbots in their everyday lives, which renders it crucial to systematically study how people perceive and evaluate them. For companies, implementing chatbots can lead to cost savings, but requires first investments in skilled personnel and technology (Vial et al., 2023). Our study can provide guidance to companies and practitioners on how to design efficient, accurate, and accessible chatbots that can improve the customer’s service experience while being economically beneficial to the company.

Chatbots are automated text-based dialogue systems with which humans can interact via natural language (Shawar & Atwell, 2007). One can communicate with chatbots and not only through them, which makes them communicative subjects (Guzman, 2018). Many customer service chatbots rely on AI-based technologies such as natural language recognition and processing. Although labeled the gold standard in human-chatbot interaction, free text interactions with chatbots often result in frustration and abandoned conversations because the chatbot does not understand user queries, phrases, or context (Klopfenstein et al., 2017). In addition, free text interactions require users to input text through typing on their keyboards, which can be a cumbersome and unattractive task, particularly for users who are not experienced at typing. Structuring elements, such as buttons that display pre-defined answer options for users to click on, are already employed in state-of-the-art chatbots as well as LLM-powered chatbots, e.g., Microsoft’s Bing chatbot (Bing, 2023). Buttons make the conversation flow more transparent but can also negatively affect perceived humanness, social presence, and user-chatbot bonding (Diederich et al., 2019; Mai et al., 2022). In this study, we thus want to investigate how the interaction mechanism (buttons vs. free text) affects the perception of the chatbot.

Most people use chatbots for productivity reasons, i.e., because they are easy to use and respond to requests quickly (Brandtzaeg & Følstad, 2017). However, chatbots that do not meet these requirements might negatively impact customers’ trust and word-of-mouth (Seeger & Heinzl, 2021). Since non-understandings are common in user-chatbot conversations, it is crucial to prevent users from losing motivation and abandoning the conversation (Ashktorab et al., 2019; Følstad & Taylor, 2020). How the error response is presented, i.e., whether the chatbot apologizes and reveals its weaknesses, is a potential factor in improving user perception but has been neglected in research on chatbots so far (Benner et al., 2021). We, therefore, want to study how a social response to errors affects users’ perceptions of the chatbot, satisfaction, use intention, and emotional connection with the company.

The aim of our paper is to answer the following research question: How do free text interaction and social error responses influence perceived warmth and competence, satisfaction, intention to use, and emotional connection with the company? We use the Media Are Social Actors (MASA) paradigm (Lombard & Xu, 2021), which extends the Computers Are Social Actors (CASA) paradigm (Nass et al., 1994), social response theory (Lombard & Xu, 2021; Nass & Moon, 2000) and the concept of anthropomorphism (Epley et al., 2007) to derive our hypotheses. We test our hypotheses using an online experiment where participants interact with a state-of-the-art chatbot in a typical online retail customer service scenario.

Previous studies have investigated either the effects of a chatbot’s interaction mechanism (Diederich et al., 2019; Haugeland et al., 2022) or the effects of a chatbot’s verbal cues (Chattaraman et al., 2019; Lee et al., 2020) without considering possible interaction effects between the two. We complement existing research on the impact of combinations of different types of social cues on user responses (Lombard & Xu, 2021) by examining whether interaction mechanism and a chatbot’s reaction to errors interact in their effects on the outcomes of interest. In addition to the theoretical implications, studying this interaction may also have important practical implications for the design and implementation of human-like and efficient customer service chatbots by practitioners and companies. By analyzing whether two dimensions central to the evaluation of new agents, i.e., warmth and competence, mediate the effects of interaction mechanism and error responses on the outcome variables, we expand existing research by shedding light on the mechanisms underlying the relationship between social cues and user responses.

This study extends previous research on the impact of a chatbot’s interaction mechanism (Diederich et al., 2019; Haugeland et al., 2022) and error handling (Ashktorab et al., 2019; Lee et al., 2010) by investigating their influence on five user-related outcomes. The five outcomes represent central variables for assessing user perceptions, user experiences, and the company perspective (Cheng & Jiang, 2020; Fiske et al., 2007; Verma et al., 2016). First, we look at the warmth and competence of the chatbot as perceived by users. Warmth and competence are the basic dimensions along which people evaluate new actors (Fiske et al., 2007). Based on prior research on AI-based agents (Blut et al., 2021; Cameron et al., 2021), we argue that more human-like interactions with a chatbot might be positively correlated with users’ evaluations thereof. We do not include perceptions of agents that have been studied extensively, such as social presence or trust (Zierau et al., 2020). Negative emotions like anger (Crolic et al., 2022) are not considered either because the interaction in this study is unlikely to elicit much anger due to the experimental design, i.e., all participants have to solve the same fictitious tasks with the help of the chatbot. Second, we examine two central variables related to users’ experiences with the system: satisfaction, i.e., the extent to which the chatbot experience meets their needs and expectations (ISO, 2018), and intention to use the system, i.e., the desire to use it in the future (Gopinath & Kasilingam, 2023). Satisfaction with a system is considered one of the main objectives of system design and evaluation in human-machine interaction (ISO, 2018). Both variables are commonly used to investigate user responses to machine agents in human-computer interaction research (Cheng & Jiang, 2020; De Andrés-Sánchez & Gené-Albesa, 2023). Third, we consider a company-related outcome, namely users’ emotional attachment to it, i.e., the affinity between consumers and the company. Previous research has shown that anthropomorphic chatbot cues positively impact the relationship-building of companies with their customers (Araujo, 2018), which in turn has been shown to increase word-of-mouth, customer loyalty, and continuity expectation (Verma et al., 2016).

1. Theoretical framework

1.1. Social responses to chatbots

An established framework for researching human-machine communication is the Computers Are Social Actors (CASA) paradigm. The CASA paradigm was derived from the media equation, which states that humans treat computers and other media as real people (Reeves & Nass, 1996). According to the CASA paradigm, humans apply similar rules in their interactions with machines as in their interactions with other humans (Nass et al., 1994; Nass & Moon, 2000). The idea of humans treating machines as social actors is closely linked to the concept of anthropomorphism, the tendency of humans to ascribe human characteristics, motivations, intentions, and emotions to non-human entities (Epley et al., 2007). Nass & Moon (2000) propose three main characteristics incorporated in machines leading people to act socially towards them. These properties are especially evident in text-based chatbots. The first characteristic refers to chatbots outputting written or spoken words. Second, chatbots are based on dialogue and turn-taking and are thus highly interactive. Third, chatbots fill roles traditionally occupied by humans, e.g., customer service agents, learning tutors, or mental health advisors (Følstad et al., 2019). These characteristics make chatbots media agents, defined as “technological artifacts that demonstrate sufficient social cues to indicate the potential to be a source of social interaction” (Gambino et al., 2020, p. 73).

The Media Are Social Actors (MASA) paradigm structurally extends the CASA paradigm by emphasizing the impact of social cues on users’ responses to various kinds of media agents like social robots and voice- or text-based conversational agents (Lombard & Xu, 2021). Media agents provide verbal and non-verbal social cues that send out social signals, which in turn elicit social user responses (Feine et al., 2019; Go & Sundar, 2019; Lombard & Xu, 2021). Social cues are defined as “features salient to observers because of their potential as channels of useful information” (Fiore et al., 2013, p. 2). In contrast, social signals refer to how the perceiving person interprets the social cues of the sender media agent (Fiore et al., 2013). Examples of social cues include eye gaze, gesture, body movement, and language use, which can send out social signals like responsiveness, interactivity, and even personality and social identity (Feine et al., 2019; Lombard & Xu, 2021).

Social cues in media agents have previously been categorized in different ways. For example, Seeger et al. (2021) distinguish between human identity, verbal, and non-verbal cues, whereas Feine et al. (2019)’s taxonomy comprises verbal, visual, auditory, and invisible cues. This study investigates the effect of a chatbot’s verbal cues in the form of its responses to errors on user responses. Verbal cues are expressed in written or spoken words, which can signal responsiveness, attentiveness, understanding, support, and human-likeness (Feine et al., 2019; Lombard & Xu, 2021). Previous research has shown that we are more easily persuaded when we perceive a conversational partner to be similar vs. not similar in how we communicate verbally (Fogg, 2003). Even slight verbal variations, e.g., regarding the conversational style of the chatbot or short text messages, e.g., “I am sorry,” can convey human-likeness and, thus, influence our perceptions and evaluations of media agents (Van Pinxteren et al., 2023; J. Chen et al., 2023; Chattaraman et al., 2019). However, as our study not only examines verbal error responses but also different mechanisms of interaction between user and chatbot, we also make use of Fogg (2003)’s category of social dynamics cues. Social dynamics cues are “unwritten rules of how people interact with each other” (Fogg, 2003, p. 105) and entail turn-taking, cooperation, and reciprocity (Lombard & Xu, 2021). Cues representing social dynamics follow from the mechanism through which the interaction with a media agent takes place, e.g., through free text input or by clicking on buttons with pre-defined answer options (Jain et al., 2018). Social dynamics cues can thus influence the perceived engagement and interactivity of a media agent, which in turn can affect user responses (Haugeland et al., 2022). In the following two sections, we lay out the current state of the research and present our hypotheses on the impact of interaction mechanisms and error responses on user responses.

1.2. Interaction mechanism and user responses to chatbots

A factor that might strongly influence user acceptance is the interaction mechanism (Haugeland et al., 2022), also referred to as message form (Klopfenstein et al., 2017), interaction paradigm (Valério et al., 2020), interaction method (Mai et al., 2022) or turn-taking mechanism (Zierau et al., 2020). The interaction mechanism of a system refers to the kind of input it receives from the user during the interaction (Jaimes & Sebe, 2007). Different types of inputs are made in different ways: In this article, we distinguish between free text interaction, i.e., interaction with a chatbot by entering text via the keyboard, and button interaction, i.e., interaction with a chatbot by clicking on buttons with pre-defined answer options using the mouse.

Previous research on the effects of chatbots’ interaction mechanisms on user responses has yielded mixed results. On the one hand, there is empirical evidence that using buttons reduces a chatbot’s perceived humanness, potentially leading to lower user acceptance. In Diederich et al. (2019)’s experiment, for example, participants interacted with one of two chatbots where one was based on free text while the other was based on a combination of free text and preset answer buttons. The authors found that buttons reduced the perceived humanness and social presence of the conversational agent in the interaction. Along these lines, the experimental study by Mai et al. (2022) showed that free-text interaction improves the working alliance by enhancing the feeling of mutual respect, perceived sympathy, and appreciation by a coaching chatbot. A mixed-methods case study by Valério et al. (2020) showed that people with scientific backgrounds do not prefer free text over button chatbots. In turn, data from daily chatbot users analyzed by Jain et al. (2018) revealed that users preferred chatbots “that provided either a “human-like” natural language conversation ability or an engaging experience that exploited the benefits of the familiar turn-based messaging interface” (p. 895).

On the other hand, structured chatbot designs utilizing buttons might enhance user experience (Narducci et al., 2020). In an experiment on voting advice applications, Kamoen et al. (2022) found that buttons appeared more playful to all participants and easier to use to lower-educated participants than text fields. A randomized within-subjects experiment by Haugeland et al. (2022) found that button interaction leads to a higher pragmatic and hedonic user experience than free text interaction.

Based on the MASA paradigm and social response theory, we argue that users perceive interactions via free text as more human-like, i.e., more like their communication with other humans, and, accordingly, to have more similar social dynamics than interactions via buttons. Thus, users perceive chatbots based on free text input as warmer. Based on higher similarity to human-to-human interactions, we also propose that chatbots based primarily on free text input are perceived as more competent as they seem able to understand the requests written by the user. In addition, free text interaction mimics natural human communication more adequately than button interaction, leading to users expending less cognitive effort (Gambino et al., 2020) and, in turn, to increased satisfaction, intention to use it, and emotional connection with the company.

H1. When users interact with a chatbot primarily via free text (as opposed to buttons), (a) perceived warmth and (b) competence of, (c) satisfaction with, (d) intention to use the chatbot, and (e) emotional connection with the company will be higher.

1.3. Responses to errors and user acceptance

As conversational breakdowns in user-chatbot conversations occur frequently, users should be prevented from losing motivation and abandoning the conversation (Ashktorab et al., 2019; Følstad & Taylor, 2020). Investigating the error repair strategies of chatbots is thus crucial. Repair strategies refer to strategies “that a chatbot […] could adopt to tackle the above problems—providing evidence for the breakdown and supporting repair towards a desirable direction for the system model” (Ashktorab et al., 2019, p. 2).

Ashktorab et al. (2019) found proactive repair strategies, like providing options and explanations, were favored among several chatbot repair strategies. Sheehan et al. (2020) similarly found that a chatbot that tries to clarify its mistake increases people’s adoption intent compared to a chatbot that ignores its mistake. Positive effects of social error-handling strategies, such as offering an apology after an error to mitigate its negative impact, have also been demonstrated in social robots (Cameron et al., 2021; M. K. Lee et al., 2010). Besides apologizing for mistakes, social error-handling strategies include justifying or explaining the reasons for the error, promising to do better in the future (Robinette et al., 2015), as well as self-blame (Groom et al., 2010).

In particular, combinations of different error management strategies can lead users to continue the conversation (Benner et al., 2021). The combination of apologizing to the user for the error and admitting weaknesses has great potential to prevent conversation dropouts but has not been systematically studied for chatbots. We define an error response that contains verbal human-like cues like apologizing and admitting weaknesses as a social error response. A neutral response to errors does not contain verbal social cues. We argue that a social error response is more efficient in increasing our outcomes of interest than a neutral error response because people perceive it to be more human-like. We propose the following hypothesis:

H2. When a chatbot’s responses to errors are social (as opposed to neutral), (a) perceived warmth and (b) competence of, (c) satisfaction with, (d) intention to use the chatbot, and (e) emotional connection with the company will be higher.

1.4. The interplay of interaction mechanism and error response

In line with the MASA paradigm and social response theory, social error responses in combination with free text (vs. button) input might make the chatbot appear more human-like as more social cues lead to more positive user responses (Lombard & Xu, 2021). Chatbots based on pre-defined buttons are perceived as more rule-based and less error-prone due to their structured design (Jain et al., 2018; Valério et al., 2020). However, suppose a button-based chatbot does not understand a request. In that case, users might expect a neutral and formal response rather than a social one, which might be more likely perceived as inconsistent, resulting in feelings of irritation and confusion among users. User responses might thus be less positive, as research on combining text-based chatbots with a human-like avatar (Ciechanowski et al., 2019) has shown. We do not expect substantial differences in user responses between free text and buttons when the error response is neutral since neutral error responses do not contribute as much to a human-like perception of the chatbot as social ones. Therefore, we hypothesize the following:

H3. The type of interaction mechanism moderates the effect of the error response on the outcomes, such that the effect of social (as opposed to neutral) error responses is stronger when the interaction mechanism is free text (as opposed to buttons).

1.5. Underlying processes

When people are confronted with a new actor, they evaluate them on two basic dimensions: perceived warmth and competence (Fiske, 2018). According to the MASA paradigm and Social Response Theory, people treat technological entities, including chatbots, as social actors (Nass & Moon, 2000). We, therefore, argue that people evaluate a chatbot they encounter for the first time similarly to another person. An actor’s first impression, e.g., regarding their likability or perceived intelligence, is often the most influential factor for their further assessment—and if positive, can lead to more positive evaluations (Bartneck et al., 2009). Previous research has shown that the more human-like, and therefore, similar to oneself, a machine is perceived, the more people tend to like and be socially attracted to it, and the more intelligent people perceive it (Beattie et al., 2020; Ciechanowski et al., 2019; Toader et al., 2019). Consequently, when people perceive a machine to be likable and intelligent, user responses like the intention to use the machine turn out to be more positive (Blut et al., 2021; Cameron et al., 2021). Perceived intelligence refers to the extent to which people believe a machine is capable of independent thought, learning, and problem-solving; likability refers to qualities that elicit a favorable (first) impression (Bartneck et al., 2009). Free text interaction and social error responses are social cues that make the conversation with the chatbot more human-like. As H1 and H2 suggest positive effects of free text interaction and social error responses on satisfaction with, intention to use the chatbot, and emotional connection with the company, we assume these effects are mediated by perceived warmth and competence.

H4.1. Perceived warmth mediates the positive effects of free text interaction on (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company.

H4.2. Perceived competence mediates the positive effects of free text interaction on (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company.

H5.1. Perceived warmth mediates the positive effects of social responses to errors on (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company.

H5.2. Perceived competence mediates the positive effects of social responses to errors on (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company.

The research framework of this study is shown in Figure 1.

Figure 1. Research framework.

Conceptual model showing how the study’s independent, mediator and dependent variables are connected.

2. Method

2.1. Overview

We conducted a 2 (interaction mechanism: buttons vs. free text) x 2 (error response: neutral vs. social) between-subjects online experiment. After providing informed consent, participants were randomly assigned to one of four conditions where they interacted with a real customer service chatbot that we adapted to this specific research setting in collaboration with a company specializing in Conversational AI to make the setting as realistic as possible. We chose this context as chatbots are already frequently employed in customer service and continue to hold great potential in this area (Costello & LoDolce, 2022). Each condition comprised between 93 and 110 participants. The study was approved by the ethics committee of the Leibniz-Institut für Wissensmedien Tübingen (LEK 2022/046) and preregistered at https://osf.io/shb6z.

2.2. Participants

An a-priori power analysis to determine the minimum sample size to detect an effect of f = 0.14, with α = .05 and a power (1 − β) of 80% yielded 403 participants (Faul et al., 2007). To account for potential exclusions, we recruited 450 German participants via the online sampling platform Prolific in October 2022. Respondents received £2.85 for participating in the experiment, which took around 20 minutes.

Following our preregistration, we excluded participants who had not followed the experimental instructions (n = 30) or experienced technical difficulties and could not interact with the chatbot (n = 4). The final sample comprised 416 cases. Participants’ average age was 29.90 years (SD = 9.16, range = 18, 71), and 50.00% identified as females. 61.78% indicated having used a chatbot at least once in the last three months.

2.3. Procedure

After starting the questionnaire, participants were asked to put themselves in a typical customer service scenario. They should imagine that they have placed an order and now have some questions to be answered and tasks to be completed by the chatbot. All participants received the same set of tasks, i.e., they were to find out (1) the balance of the gift card used to pay, (2) what happens if the recipient is not at home when the package is delivered, (3) the current location of the package, (4) the cost of return shipping, and they were (5) to subscribe to the newsletter (optional). To ensure anonymous data collection, all participants were given the same fake customer credentials, such as an email address, order number, parcel tracking id, and pin code, to share with the chatbot while working through the list of questions and tasks. We gave participants some general instructions for chatbot interaction: type their requests as free text in the text box if no options were offered, try to rephrase their request if the chatbot did not understand it, and type “I want to start over” if they wanted to start a new conversation. After reading the instructions, participants were prompted to click a button, opening a chat window embedded in the online survey. To make it easier for participants to answer the questions, they could see the list of tasks and the credentials while interacting with the chatbot. Participants received instructions to interact with the chatbot for a maximum of five minutes. To make the conversation more realistic and to be able to manipulate the experimental factor error response later, participants experienced at least one situation where the chatbot did not understand the user’s request during the interaction. We chose this type of error because failure to understand user requests is one of the most common mistakes users encounter when interacting with a chatbot (De Sá Siqueira et al., 2023). After the interaction, participants answered a questionnaire on their perceptions of the chatbot and the interaction. Participants could then comment on the study and were redirected to Prolific.

2.4. Stimuli and manipulation of independent variables

To give our research practical relevance and increase external validity, we created the chatbots in collaboration with a company specialized in conversational AI. The partner provided access to their low-code Conversational AI platform, which allowed us to create four chatbot versions for experimental purposes. The chatbots were based on a set of technologies that power automated human-like interactions, e.g., natural language processing (NLP), which enables the chatbots to understand human language text, and the deep learning architecture BERT¹ which represents the key functionality of the matching process between the intent predicted from a user’s chat message and the chatbot’s response. The conversations with the chatbots were largely automated, except for the error situations, which we pre-defined (Appendix A). Specifically, when participants tried to solve tasks 2 (what happens if the recipient is not at home when the package is delivered) and 4 (the cost of return shipping), they were automatically redirected to separate conversational flows. No matter what they entered here, the chatbot did not understand their request and asked them to rephrase their question. After participants had rephrased, the chatbot provided the correct answer. As the interaction with the chatbot took place freely and unsupervised, some participants might have encountered more than two errors.

Anthropomorphic cues improve user responses to text-based chatbots, e.g., emotional connection with the company (Araujo, 2018) and user compliance (Adam et al., 2021). We thus equipped our chatbots with anthropomorphic cues, i.e., a gender-neutral name (“Mika”) and profile picture. The chatbots resembled currently employed customer service chatbots in design and functionality (Clustaar, 2019). The chatbots greeted the participants in all conditions, saying, “Hello! I am Mika, the chatbot of myclothes.deu”. Prior research has shown that dynamically delayed response times increase user satisfaction (Gnewuch et al., 2018). Thus, the chatbot typically responded after two seconds and after three seconds if its response was longer than two sentences or if the chatbot had a task to complete, such as registering the user for the newsletter.

2.4.1. Interaction mechanism

Interaction mechanism was manipulated by creating two chatbot versions: One chatbot required participants to enter their requests via their keyboard into the text field. The other chatbot was based on pre-defined buttons, i.e., participants had to click the respective button to make their requests. No free text input was possible when participants could select a button. The chatbots were based either mainly on free text or mainly on buttons. Since there is no pure type in the real world, the free text chatbot also had a button menu at the beginning, and the button chatbot also had two free text input situations where participants had to enter the fake credentials we provided, such as their gift card pin code or newsletter signup email address.

To test the chatbots’ functionality, we conducted a two-group (button vs. free text chatbot) pre-study with 50 subjects recruited via Prolific (n_Female = 24, M_Age = 29.6, SD_Age = 10.20). In addition to common usability scales, such as the User Experience Questionnaire (UEQ; Schrepp et al., 2017), the System Usability Scale (SUS; Bangor et al., 2008), and the Chatbot Usability Questionnaire (CUQ; Holmes et al., 2019), participants were asked open-ended feedback questions about positive and negative aspects of the chatbot as well as selected demographic information. The pretest showed that both chatbots worked reliably. Usability scores were in the acceptable range. UEQ was measured on a scale from −3 to +3 where values between −0.8 and 0.8 represent a neutral evaluation of the corresponding scale, values above 0.8 a positive, and values below −0.8 a negative evaluation (Schrepp et al., 2017): M_Button = 1.06, SD_Button = 0.86, M_{Free text} = 0.67, SD_{Free text} = 0.85. A two-sample t-test yielded no significant differences between the two conditions (t(46.95) = 1.59, p = .118). We assessed SUS and CUQ on scales from 1 = fully disagree to 5 = fully agree and converted the results to an overall score (0–100). According to the interpretation scheme (Bangor et al., 2008), both chatbots yielded acceptable SUS scores (M_Button = 83.20, SD_Button = 11.10, M_{Free text} = 75.25, SD_{Free text} = 12.00) with the button chatbot being slightly more acceptable, as shown in a two-sample t-test (t(46.32) = 3.40, p = .020). CUQ scores were similar in magnitude and direction (M_Button = 76.75, SD_Button = 11.90, M_{Free text} = 69.16, SD_{Free text} = 10.1, t(46.32) = 2.41, p = .020). Based on the participants’ comments, we made minor improvements to both chatbots, e.g., regarding knowledge base and category naming. Figure 2 shows exemplary screenshots of the button and free text chatbots used in the experiment.

Figure 2. Translated exemplary screenshots of button (left) and free text chatbots.

Two screenshots of a written chat dialogue between a user and a chatbot. One interaction takes place via buttons, the other via free text.

2.4.2. Error response

After ensuring that the button and free text chatbots functioned reliably and that each conversational flow included two situations where the chatbot did not understand the user’s request, we focused on manipulating the chatbots’ error response, our second experimental factor. In the social error response conditions, the chatbot responded to a mistake it made socially, i.e., its response comprised an apology and a disclosure statement that it is still learning: “I am sorry, unfortunately, I did not understand that. I am still learning. Please rephrase your request”. In the neutral error response conditions, the chatbot responded to errors neutrally, i.e., it simply said: “I did not understand that. Please rephrase your request”. We provide translated exemplary conversations for each condition in Appendix A. Videos of conversations in the original language are available in our OSF repository: https://osf.io/j8hde/.

2.5. Measures

2.5.1. Perceived warmth and competence

Perceived warmth refers to a person’s perceived friendliness, likability, and sociability; perceived competence refers to a person’s perceived intelligence, capability, and agency (Fiske, 2018). To assess users’ perceptions of the chatbot’s warmth and competence, we used the likability and perceived intelligence scales from the German version of the Godspeed Questionnaire (Bartneck et al., 2009; Cameron et al., 2021), comprising five items each. There are alternative scales that emphasize other facets of perceived warmth and competence, e.g., Lee et al. (2023) operationalized perceived warmth and competence using scales initially developed to measure subscales of trusting beliefs (McKnight et al., 2002) and Zheng et al. (2023) used a perceived warmth scale that also captures the perceived sociability during the interaction. However, warmth and likability as well as competence and intelligence, respectively, are conceptually very similar cognitive responses to new actors (Cameron et al., 2021; Fiske et al., 2007; Wojciszke et al., 2009). In addition, the items of the Godspeed questionnaire scales were explicitly developed to assess key concepts of human-robot interaction and are commonly used in HCI research (Weiss & Bartneck, 2015). Therefore, participants were asked to rate their impressions of the chatbot on seven-point semantic differential scales. Likability items included “unlikable–likable,” “unfriendly–friendly,” “impolite–polite,” “agreeable–disagreeable,” and “awful–nice”. Perceived intelligence items included “incompetent–competent,” “ignorant–educated,” “irresponsible–responsible,” “unintelligent–intelligent,” and “unreasonable–reasonable”.

2.5.2. Satisfaction

We assessed satisfaction with the chatbot using three items by Cheng & Jiang (2020). The items, i.e., “I am satisfied with the chatbot,” “The chatbot did a good job,” and “The chatbot did what I expected,” were translated to German, adapted to chatbots, and measured on 7-point Likert-type rating scales (1 = do not agree at all, 7 = completely agree).

2.5.3. Intention to use

Intention to continuously use the chatbot was measured using a validated scale by Cheng & Jiang (2020), comprising three items, i.e., “I would continue to use the chatbot,” “I would use the chatbot for purposes other than my current use,” and “I would explore other services of the chatbot than the ones I currently use”. Continued use was translated to German, adapted to chatbots, and measured on a 7-point Likert-typed rating scale (1 = do not agree at all, 7 = completely agree).

2.5.4. Emotional connection with the company

Emotional connection with the company refers to “the affinity between customers and [a] brand” (Christodoulides et al., 2006, p. 805). We adapted Christodoulides et al. (2006)’s three-item scale by leaving out the item “I feel related to the type of people who are [X]’s customers” and replacing “brand” with our company name in the other two items because we were interested in the emotional connection with the company (Araujo, 2018). The items “I feel like myclothes.deu actually cares about me” and “I feel that myclothes.deu really understands me” were translated to German and measured on 7-point Likert-typed rating scales (1 = do not agree at all, 7 = completely agree).

Descriptive statistics, bivariate correlations, and Cronbach’s alpha values for all study variables are displayed in Table 1.

Table 1. Descriptive statistics, correlations, and alpha values for study variables.

Variable	M	SD	α	1	2	3	4	5
1.  Perceived warmth	5.43	1.05	.93	–
2.  Perceived competence	4.87	1.05	.87	.67	–
3.  Satisfaction	5.03	1.44	.93	.61	.74	–
4.  Intention to use	4.20	1.60	.88	.55	.63	.74	–
5.  Emotional connection with the company	4.16	1.48	.92	.60	.70	.59	.64	–

Note. N = 416. Cronbach’s α. All correlations p < .001.

2.5.5. Manipulation checks

We included manipulation checks for interaction mechanism and error response to ensure the manipulations were effective. To ensure participants encountered at least one chatbot error, we asked them how many mistakes they experienced during the interaction. To assess the error response manipulation, we asked participants whether the chatbot apologized for its error and disclosed that it was still learning at some point during the interaction.

3. Results

3.1. Manipulation checks

The analysis of conversation protocols and self-reported data confirmed that all participants in the final sample experienced at least one error. Most participants indicated having encountered one or two errors (61.1%), while the others encountered three or more. Most participants in the social error response conditions noticed the chatbot’s social error response; 9 participants (2.2%) did not notice the chatbot’s apology, and 57 participants (13.7%) did not notice the chatbot’s disclosure that it was still learning.

Of all participants, the majority (n = 278) solved either four or five of five tasks, 87 solved three, 28 participants solved two, and 13 participants solved one task. The tasks that were solved most often were task 1 “gift card balance” (n = 403), and 3 “parcel tracking” (n = 364). Tasks 2 “delivery issue” and 4 “return cost” with the inbuilt error as well as the voluntary task 5 “newsletter registration” were solved less often (n_Task2 = 334, n_Task4 = 284, n_Task5 = 222). Participants mostly needed one attempt to solve tasks 1 (95.53%), 3 (83.79%), and 5 (96.40%), and two attempts for tasks 2 (61.68%) and 4 (72.89%).

3.2. Hypothesis testing

To test H1–H3, we conducted five two-way analyses of variance (ANOVA) with interaction mechanism and error response as independent variables, including their interaction. We added warmth (a), competence (b), satisfaction (c), intention to use (d), and emotional connection with the company (e) as respective dependent variables (Table 2). Table 3 shows the means and standard deviations of the outcome variables by interaction mechanism and error response.

Table 2. Results of two-way analyses of variance in study variables.

Effect	Perceived warmth		Perceived competence		Satisfaction		Intention to use		Emotional connection with the company
	F ratio	η^2partial	F ratio	η^2partial	F ratio	η^2partial	F ratio	η^2partial	F ratio	η^2partial
Interaction mechanism	12.20***	.03	22.67***	.05	36.23***	.08	13.84***	.03	4.92°	.01
Error response	3.34°	.01	2.18	.01	4.89*	.01	2.65	.01	1.06	.003
Interaction mechanism × Error response	0.34	.001	0.21	.001	0.07	.0002	0.04	.0001	5.66*	.01

Note. N = 416. Interaction mechanism: 0 = button and 1 = free text. Error response: 0 = neutral and 1 = social. df = 1,412.

° p < .10; *p < .05; **p < .01; ***p < .001.

Table 3. Means and standard deviations of interaction mechanism and error response main effects on study variables.

Variable	Interaction mechanism				Error response
	Button		Free text		Neutral		Social
	M	SD	M	SD	M	SD	M	SD
Perceived warmth	5.62	0.98	5.26	1.08	5.35	1.06	5.52	1.03
Perceived competence	5.12	0.95	4.65	1.09	4.81	1.07	4.95	1.02
Satisfaction	5.46	1.21	4.64	1.53	4.90	1.52	5.18	1.34
Intention to use	4.50	1.56	3.93	1.60	4.09	1.63	4.33	1.57
Emotional connection with the company	4.33	1.43	4.01	1.52	4.09	1.40	4.23	1.57

Note. N = 416. Interaction mechanism: 0 = button and 1 = free text. Error response: 0 = neutral and 1 = social.

Contrary to our first hypothesis, we did not find positive main effects of interaction mechanism (free text) on perceived warmth and competence, satisfaction, and intention to use. Instead, participants indicated statistically significant lower values in the free text (vs. button) conditions on all dependent variables. Thus, H1 was not supported.

In line with our second hypothesis, we found a significant positive effect of social error response on satisfaction (F(1, 412) = 4.89, p = .028, η²_partial = .01). Interacting with a socially responding chatbot led to significantly higher satisfaction (M = 5.18, SD = 1.34) than interacting with a neutrally responding chatbot (M = 4.90, SD = 1.52). The effect of social error response on perceived warmth was descriptively positive but only marginally significant (M_Neutral = 5.35, SD_Neutral = 1.06, M_Social = 5.52, SD_Social = 1.03; F(1, 412) = 3.34, p = .068, η²_partial = .01). Although descriptively positive, error response had no significant effects on intention to use and emotional connection with the company. Only H2c was supported.

Regarding our interaction hypothesis (H3), we found one statistically significant interaction effect on emotional connection with the company (F(1, 412) = 5.66, p = .018, η²_partial = .01), which was, however, not in line with our expectations (Figure 3). We expected the effect of social (as opposed to neutral) error responses on the outcome variables to be stronger when the interaction mechanism is free text (as opposed to buttons). However, post hoc testing using Tukey’s HSD only yielded one significant mean difference: Participants in the social error response conditions indicated a higher emotional connection with the company when interacting with the button-based chatbot (M = 4.60, SD = 1.46) compared to the free text chatbot (M = 3.92, SD = 1.60, p = .006, d = −0.44). No difference was found in the neutral conditions (M_Button = 4.09, SD_Button = 1.36, M_{Free text} = 4.10, SD_{Free text} = 1.45, p = 1.000). H3 was thus not supported.

Figure 3. Interaction plots of error response and interaction mechanism on study variables.

Note. N = 416.

Five interaction plots showing the interaction effects of error response and interaction mechanism on perceived warmth and competence, satisfaction, intention to use, and emotional connection with the company.

To test H4 and H5, we ran separate mediation models for each outcome variable with interaction mechanism and error responses as independent variables, using the R mediation package v4.5.0 (Tingley et al., 2019). The mediation analyses conducted to test H4.1, which stated that perceived warmth mediated the effects of interaction mechanism on satisfaction, intention to use, and emotional connection with the company, yielded some interesting results (Figure 4). The interaction via free text was perceived as less warm, which led to less satisfaction, intention to use, and emotional connection with the company. Besides this indirect effect, there was still a direct effect indicating that warmth only partially explained the relationship. The same pattern emerged for the interaction mechanism effect on intention to use via perceived warmth, where a significant negative indirect effect emerged as well. Including the mediator in the model led to a negative direct effect that was smaller in magnitude than the total effect. Although there was no direct effect, we found an indirect effect from interaction mechanism on emotional connection with the company.

Figure 4. Mediation results of interaction mechanism on satisfaction, intention to use, and emotional connection with the company via perceived warmth.

Note. Interaction mechanism: 0 = button and 1 = free text. N = 416.

Mediation plot showing the results of the mediation of interaction mechanism on satisfaction, intention to use, and emotional connection with the company via perceived warmth.

Figure 5 shows the results of the mediation analyses to test H4.2. Regarding perceived competence as mediator of the relationship between interaction mechanism and satisfaction, a significant negative indirect effect emerged, with the direct effect still being significant, indicating that perceived competence only partially explained the relationship between interaction mechanism and satisfaction. A full mediation was observed for intention to use and emotional connection with the company, where we found significant negative indirect effects as well as non-significant direct effects.

Figure 5. Mediation results of interaction mechanism on satisfaction, intention to use, and emotional connection with the company via perceived competence.

Note. Interaction mechanism: 0 = button and 1 = free text. N = 416.

Mediation plot showing the results of the mediation of interaction mechanism on satisfaction, intention to use, and emotional connection with the company via perceived competence.

A partial mediation with marginally significant indirect and total effects emerged when including perceived warmth as mediator of the relationship between error response and satisfaction (H5.1). We computed mediation models for the other outcomes, although no significant total effects were found when testing H2. No significant indirect effects were found for the other outcomes. Also, no significant indirect effects were found when including perceived competence as a mediator (H5.2). The full mediation models for H5.1 and H5.2 are displayed in Appendix B.

An overview of the results can be found in Table 4.

Table 4. Overview of results.

Hypothesis	Result
H1. Interaction mechanism → (a) perceived warmth and (b) competence of, (c) satisfaction with, (d) intention to use the chatbot, and (e) emotional connection with the company	Rejected; reverse effects
H2. Error response → (a) perceived warmth and (b) competence of, (c) satisfaction with, (d) intention to use the chatbot, and (e) emotional connection with the company	Partially confirmed (H2c)
H3. Interaction mechanism × error response → (a) perceived warmth and (b) competence of, (c) satisfaction with, (d) intention to use the chatbot, and (e) emotional connection with the company	Rejected
H4.1. Interaction mechanism → perceived warmth → (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company	Rejected; significant indirect effects, but reverse direction
H4.2. Interaction mechanism → perceived competence → (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company	Rejected; significant indirect effects, but reverse direction
H5.1. Error response → perceived warmth → (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company	Rejected
H5.2. Error response → perceived competence → (a) satisfaction with, (b) intention to use the chatbot, and (c) emotional connection with the company	Rejected

3.3. Additional analyses

As the pretest results indicated, perceived ease of use might be a potential influencing factor of users’ chatbot acceptance. Exploratory Welch two-sample t-tests yielded a higher perceived ease of use among participants who interacted via buttons compared to free text (M_Button = 5.41, SD_Button = 1.20, M_{Free text} = 4.74, SD_{Free text} = 1.49; t(406.90) = 5.03, p < .001, d = 0.49). Participants who interacted via buttons (vs. free text) also indicated fewer encountered errors (M_Button = 1.89, SD_Button = 0.74, M_{Free text} = 3.31, SD_{Free text} = 2.03; t(276.58) = −9.65, p < .001, d = −0.92). When controlling the relationship between interaction mechanism and the outcome variables for perceived ease of use, the negative effects of the free text interaction were mitigated: The effects on warmth, competence, intention to use, and emotional connection with the company became non-significant. The effect on satisfaction became less negative. Full results of the additional analyses can be found in our OSF repository.

4. Discussion

The rapid development of AI-based technologies has enabled chatbots to interact with users in a natural and human-like way. However, conversing with chatbots via free text often results in users experiencing feelings of frustration and abandoning conversations (Følstad & Taylor, 2020). To investigate the impact of interaction mechanism and error response, we conducted an online experiment where participants directly interacted with a chatbot in a common customer service setting. Four main findings emerged.

First, the free text chatbot was rated more negatively than the button chatbot regarding perceived warmth, competence, user satisfaction, and intention to use, which does not correspond with our expectations. This finding also contradicts previous research, which found free text interaction to positively impact service encounter satisfaction through feelings of humanness in the interaction (Diederich et al., 2019). Strikingly, additional analyses showed that the negative effects of interaction mechanism on the outcomes of interest were mitigated when controlled for perceived ease of use. This suggests that users are well trained through previous online experiences in interacting by mouse and hence preferred the button chatbot, which was navigable using the mouse. This is directly related to the fact that typing a request typically takes longer than simply clicking a button. Users might have preferred the button chatbot because the buttons matched the tasks very well, were easy to find, and structured the problem, whereas free text input requires the user to come up with the proper words, questions, and requests, as our exploratory analyses suggest. When users entered their request in this study, they could either do so via pre-defined buttons or via free text, i.e., only one interaction mechanism was available at the same time. Future research could investigate whether allowing users to choose between buttons and free text input would be beneficial, as this provides guidance and flexibility simultaneously (Haugeland et al., 2022).

Second, social error responses increased users’ satisfaction with the chatbot (H2). Our study conceptually replicated previous findings on the positive impact of social error responses on user satisfaction with social robots in the domain of text-based chatbots (Cameron et al., 2021). However, we did not find significant effects of social error responses on perceived warmth, competence, intention to use, and emotional connection with the company.

Surprisingly, we did not find interaction effects in line with our third hypothesis. There was, however, one significant interaction effect: The free text chatbot that responded socially to errors lead to a lower emotional connection with the company than the button chatbot that responded socially to mistakes, whereas the two chatbots did not differ when not responding socially to errors. A reason for that could be that the button (vs. free text) chatbot, which apologized for its errors and, at the same time, communicated that it is still learning, might have signaled to users that the company tries to anticipate its customers’ needs by collecting frequently asked questions and turning them into buttons. However, since we did not preregister this effect, its interpretation is exploratory and must be taken cautiously. Overall, we do not find evidence that the effects of social error responses on warmth, competence, satisfaction, and intention to use depend on how users interact with a chatbot.

Fourth, perceived warmth and perceived competence mediated the effect of interaction mechanism on the outcomes. However, mediation effects did not go in the predicted direction (H4). Instead, as suggested by our results for H1, free text interaction predicted less satisfaction, intention to use, and emotional connection with the company, presumably because participants perceived the free text chatbot to be less warm and less competent. The result of our additional analyses, namely, that perceived ease of use mitigated the negative effects of free text interaction as well as the fact that perceived ease of use was strongly correlated to both perceived warmth (r = .54) and competence (r = .59), suggest that the lower the perceived ease of use of a chatbot, the less warm and competent users perceive the chatbot. We did not find perceived warmth and competence to be significant mediators of the relationship between error response and the outcomes.

4.1. Theoretical implications

The MASA paradigm and Social Response Theory claim that the more social or human-like a chatbot is, the more positive the user responses will be (Lombard & Xu, 2021). However, this prediction appears to discount the relevance of usability aspects to the extent that they matter more than previously thought: Our results suggest that the benefits of using buttons, i.e., structuring the conversational flow and enhancing navigability, appear to outweigh the benefits of human-like interactions with chatbots. From a theoretical point of view, our findings thus challenge the applicability of the MASA paradigm in task-based contexts like customer service. There might be a difference between the effects of a human-like interaction mechanism and other human-like cues of the chatbot, e.g., language style or personality. Our study suggests that while human-like language cues generally positively impact user responses (Araujo, 2018; Adam et al., 2021), a human-like interaction mechanism might have different effects because it might be perceived as inefficient and impractical in particular in reasonably structured problems such as customer service (Haugeland et al., 2022).

The findings may also be an indication that the preference for button vs. free-text bots is contingent on the type of tasks the user has to solve. The tasks we studied could have been more suitable to be answered based on buttons (vs. free text). It would be interesting to investigate which questions users believe the button and free text bots are suitable for, i.e., buttons might be more suitable for task-related queries, and free text might be more suitable for emotional/social topics.

Although participants in this study interacted with an actual chatbot, the scenario was fictitious, i.e., the conversation was not about an actual order, reducing participant stakes. However, solving real-world tasks in the context of customer service, e.g., keeping track of a parcel, may also be associated with emotional responses, such as anger or frustration. Future research could, therefore, explore the impact of the interaction mechanism in contexts where there is more at stake for the user, i.e., where they need to explain, argue, or justify more complex matters and situations or in situations where an active exchange and social factors matter more, e.g., in mental health or financial counseling.

Our results on the effect of social error responses on satisfaction align with the main premise of social response theory, i.e., the more social cues a media agent possesses, the more positive users respond to it (Nass & Moon, 2000). We did not find significant effects of social error responses on the remaining outcomes. Language cues are secondary cues, i.e., “they have less power in activating user’s social perception and responses” (Lombard & Xu, 2021, p. 34). The apology, combined with the statement that the chatbot is still learning, may not have been salient enough to elicit feelings of similarity with the chatbot. This could be why we did not find the expected effects for perceived warmth, competence, intent to use, and emotional engagement with the company. At the same time, the wording we chose reflects what is commonly used in real chatbot applications. Hence, it appears that the social error response does not seem to be the decisive factor that governs user perceptions of chatbots. Our mediation findings indicate that perceived warmth and competence are indeed central constructs for people’s evaluations of a chatbot (Fiske et al., 2007).

4.2. Practical implications and outlook

Our findings have several implications for marketing practice (e.g., Huang & Rust, 2021). A direct corollary of the results is that firms seeking to implement chatbots may want to leverage the benefits of both interaction mechanisms, i.e., buttons can structure the conversational flow of a chatbot when the problem is well-defined, and free text input might help with new user queries or queries that occur less frequently. Hence, the findings suggest that companies should give relatively more weight to the usability aspect over the potential benefit of human-like interaction mechanisms. In contrast, firms need to be less concerned about the way they implement an error response, i.e., consumers appear to appreciate a social error response, but we do not have evidence that this is a strong driver of how they view the chatbot or the firm.

It is also of relevance for firms to understand how these findings relate to the role of the recently released LLMs like GPT-4 (OpenAI, 2023). LLMs have the potential to provide even more efficient and accurate automated customer service, e.g., by providing personalized, human-like responses to customer requests (Floridi & Chiriatti, 2020; Rivas & Zhao, 2023). In this light, we must ask ourselves if our results will look different in a few years when people have more experience using chatbots powered by LLMs. However, several reasons speak for the continued relevance of our results:

LLMs have to be trained on current and relevant customer data: Until timely data is available to train these models, they are not a real alternative for companies to current customer service chatbots (Rivas & Zhao, 2023). In our experiment, participants had to solve pre-defined tasks. However, knowledge of chatbots based on LLMs is often rather broad than deep, i.e., they lack knowledge on specific topics. ChatGPT, for example, is often used for writing poems in different styles (Haque et al., 2022) but not necessarily for specific tasks like booking a flight or registering for a degree program. Yet, the primary goal in customer service is to give customers controllable, complete, and correct information (Rivas & Zhao, 2023).

ChatGPT’s interaction mechanism is free text. However, developers of LLM-based chatbots also employ buttons; e.g., Bing’s chatbot suggests answer options and further questions via buttons (Bing, 2023). Therefore, researching the influence of different interaction mechanisms remains a relevant line of research. Even chatbots based on LLMs make mistakes or hallucinate, e.g., they give incorrect answers to math problems, make up references, or get tangled up in contradictions (Borji, 2023; Sundar & Liao, 2023; Van der Meulen, 2023). Thus, the need for an adequate response remains relevant, e.g., when a user confronts the system with an error it has made or is programmed not to answer specific questions.

Lastly, although LLMs provide many opportunities, they come with less control over the conversational flow. Our collaboration with a startup specialized in conversational AI gave us full control over our experimental stimuli, i.e., the content, timing, and type of chatbot utterances, a prerequisite for valid and reliable experimental research on human-machine communication (Greussing et al., 2022). At the same time, this high degree of control over the answers is also critical for firms that must ensure that the chatbot’s responses are always accurate (Rivas & Zhao, 2023). Hence, the lack of control over the conversational flow is not only a challenge for valid and reliable experimental studies but also for firms seeking to implement this kind of technology. This challenge is amplified by recent findings of substantial variability of LLM’s responses over time (L. Chen et al., 2023).

Chatbot quality is a potential moderator of the relationships between its social cues and users’ responses. Although the transcripts of our participants’ chat conversations indicate that this chatbot was already quite good, we see the need for future research comparing state-of-the-art chatbots with chatbots based on LLMs.

4.3. Limitations and future research

First, experimental online studies have higher internal but less external validity than field studies. We, therefore, suggest future research to validate our findings in a real-world environment. Compared to our study’s controlled experimental setting that included a fictitious scenario and fake customer credentials, customer stakes and potentially also emotions such as anger and frustration, are higher when interacting with a customer service chatbot about a real order, e.g., when a package has been lost on the way to the customer. Emotions like customer anger might moderate the effect between the predictors and user acceptance. Specifically, social cues might lose their benefits when customers already enter the interaction in an angry state (Crolic et al., 2022). In this context, qualitatively analyzing chat transcripts of interactions between customers and chatbots might provide valuable additional insights into how users talk to chatbots, how they immediately react to a chatbot apologizing for an error, and when and why users abort a conversation. Second, we investigated our research question in the customer service context, as chatbots are already commonly used here. However, there might be differences in the impact of free text interaction or social error responses regarding chatbots in domains where social and emotional factors matter more, e.g., chatbots as social companions or mental health advisors. In addition, chatbots in customer service frequently answer relatively simple and predictable questions customers might have before or after purchasing or using products or services. Future research could, therefore, address contexts in which users face more complex problems. Third, all our study participants were based in Germany. Testing our hypotheses using a sample from a different cultural context would further strengthen our results. Finally, we cannot make any statement about causality regarding the mediation analyses. A longitudinal study could provide answers to this question in future research.

5. Conclusion

This study investigated the impact of free text interaction and social error responses in human-chatbot interactions on perceived warmth and competence, satisfaction, intention to use, and emotional connection with the company through an online experiment. Interestingly, we found that chatbots using free text interaction decrease users’ warmth and competence perceptions, satisfaction, intention to use, and emotional connection with the company compared to chatbots with a structured conversation flow. Additionally, chatbots that respond to errors in a social way increase user satisfaction but not the other outcomes. Our findings suggest that designing chatbots to be more human-like is not always optimal for user perceptions of the chatbot and the employing company, particularly when the chatbot is strongly task-focused. In our study, the chatbot’s error-proneness trumped its humanness. Our insights can guide practitioners in designing efficient, accurate, and accessible chatbots.

Open science statement

Study materials, data, and corresponding R code files are available on OSF: https://osf.io/j8hde/

Acknowledgments

We thank an anonymous firm for allowing us to use and adapt their chatbot implementation. We thank Dominik Warlich for his expertise and valuable support in developing the chatbot and Natalie Kraus and Anne Bucher for their assistance in preparing the stimulus materials.

Disclosure statement

The authors report there are no competing interests to declare.

Additional information

Notes on contributors

Stefanie H. Klein

Stefanie H. Klein is a doctoral student in the Everyday Media Lab at Leibniz-Institut für Wissensmedien Tübingen, Germany. Her research focuses on human-machine communication, particularly the impact of chatbots’ conversational characteristics on user acceptance.

Dominik Papies

Dominik Papies is a Professor of Marketing at the School of Business and Economics at the University of Tübingen. His research focuses on econometric modeling of consumer and business behavior.

Sonja Utz

Sonja Utz is a Professor of Communication Via Social Media at the University of Tübingen and the head of the Everyday Media lab at Leibniz-Institut für Wissensmedien in Tübingen. Her research focuses on the effects of social and mobile media use as well as on human-machine interaction.

Notes

1 Bidirectional Encoder Representations from Transformers (BERT) is a language representation model developed by Google. BERT is used to pre-train bidirectional representations from unlabeled text. It can then be fine-tuned to create models for many language understanding tasks, e.g., question-answering and translation (Devlin et al., 2019).

Appendix A.

Exemplary Translated Conversational Flows in Button and Free Text Chatbot Conditions

Figure A. Exemplary translated conversational flows in button and free text chatbot conditions.

Note. Figures A and B show exemplary conversations with the button and the free text chatbot, respectively. Chatbot responses are shown in light grey, and user responses in dark grey.

Two flow diagrams showing exemplary translated conversational flows in the button and the free text chatbot conditions.

Appendix B.

Mediation results

Figure B1. Mediation results of error response on satisfaction, intention to use, and emotional connection with the company via perceived warmth.

Note. Error response: 0 = neutral and 1 = social. N = 416.

Mediation plot showing the results of the mediation of error response on satisfaction, intention to use, and emotional connection with the company via perceived warmth.

Figure B2. Mediation results of error response on satisfaction, intention to use, and emotional connection with the company via perceived competence.

Note. Error response: 0 = neutral and 1 = social. N = 416.

Mediation plot showing the results of the mediation of error response on satisfaction, intention to use, and emotional connection with the company via perceived competence.