Organizational Factors Affecting Successful Implementation of Chatbots for Customer Service

Abstract

While chatbots have become an important part of customer service operations, there is a knowledge gap concerning organizational aspects of chatbot implementation and management. In response to this gap, we present a study of organizational factors affecting successful chatbot implementation. The study involved six organizations that had implemented chatbots for customer service within the last three years. Interviews were conducted with chatbot project owners, managers, developers, and customer service personnel – a total of 14 interviews. Through thematic analysis, five organizational factors were detailed as important for successful chatbot implementation: (1) work and team organization, (2) change management, (3) competencies and competency acquisition, (4) organizational resources, and (5) performance measures. We also present findings on the organizations' motivations and key success criteria for chatbot implementation. Based on the findings we summarize implications for theory and practice and point out directions for future research.

Keywords:

• Chatbot
• customer service
• digital innovation
• implementation effectiveness
• innovation implementation
• organizational factors

Introduction

Chatbots powered by artificial intelligence (AI) have increasingly been taken up by the service industry since the 2016 “year of chatbot” (Dale 2016), when big tech corporations like Microsoft, Google and Amazon predicted that conversational commerce would be the next big thing. As anticipated, the commercial interest in chatbots for customer service has increased substantially in recent years (Nordheim, Følstad, and Bjørkli 2019). Gartner predicted that in 2022, 70% of global customer interactions will involve AI-based technologies including chatbots, a notable increase from 15% in 2018 (Goasduff 2020). While previous generations of chatbots for customer service were created to answer simple queries, current chatbots are designed to enrich customer experience and optimize internal operations by enabling assisted self-service, conversational commerce, intuitive onboarding, enhanced profiling, co- automated workflows, and insight mining (PSFK. 2018).

Chatbots are “machine agents with which user interact through natural language dialogue” (Følstad and Brandtzaeg 2020) and may also be referred to as dialogue systems or conversational agents. A main appeal of chatbots for customer service is their ability to provide immediate and round-the-clock service. A recent survey indicated that a substantial percentage of customers will always choose a chatbot over human customer service personnel if it saves them time (Usabilla 2018). In addition, industry reports have suggested that customers may perceive businesses that use chatbots as more innovative and efficient (PSFK 2018).

Businesses and researchers race to find the answer to “What is a good chatbot?” in order to realize the potential benefits of chatbots. The target is to design and develop chatbots that satisfy user needs and provide rewarding customer experiences. The current literature on chatbots addresses the technical aspects and features of chatbots, such as interface and interaction design (Shevat 2017) and AI-based natural language processing capabilities (Adiwardana et al. 2020), as well as user experience and preferences, such as user perceptions of human likeness and trust in chatbots (Nordheim, Følstad, and Bjørkli 2019) and preferences for chatbot visual and conversational design (Go and Sundar 2019).

Knowledge on technical aspects and user experience clearly is essential to construct “good chatbots”. However, such knowledge – addressing only the micro-level interactions between users and chatbots – may not be sufficient for chatbots to deliver the expected outcomes. Specifically, a “good chatbot” may deliver poor results due to suboptimal implementation and maintenance. Hughes, Rana, and Simintiras (2017) found that failures in information systems projects are often attributable to non-technological factors, for example poor project management. To effectively utilize new technologies organizations, a holistic “big picture” approach is needed (Winby and Mohrman 2018). All factors relevant to the implementation process need to be considered, including organization structure, task factors, environmental characteristics, and the human elements involved (Jones and Smith 2001).

While the deployment of customer service chatbots may not involve organization-wide technology overhaul, it still depends on a range of factors concerning the organization rather than the technology in itself. Hence, knowledge of the organizational factors that underpin successful implementation of chatbots for customer service is critical. However, Belanche et al. (2020) noted a scarcity of research on chatbot applications within service industries. Only a handful of studies have examined chatbots from an organizational perspective. Ivanov and Webster (2017) studied how the deployment of chatbots and other forms of artificial intelligence in the tourism industry pose practical challenges, such as resistance to change and the need to reengineer service processes. Larivière et al. (2017) examined the changing roles of employees following the introduction of chatbots and other types of AI-based support. We are aware of no research addressing organizational factors that may facilitate or impede the development and deployment of chatbots for customer service. Furthermore, there is a lack of knowledge on how organizations are impacted by chatbot implementation. In a recent literature review, Syvänen and Valentini (2020) concluded that existing research has been preoccupied with a micro-level understanding of chatbots and that research on an organizational meso-level or societal macro-level is lacking.

Driven by the gap in current knowledge, this research sets out to examine the implementation of customer service chatbots from organizational perspective. Specifically, it addresses the organizational factors impacting the successful implementation of such chatbots.

The research is based on an interview study involving six organizations that have implemented chatbots for customer service over the last three years. In total, we conduced semi-structured interviews with 14 resource persons within these organizations. Through the interviews, we acquired in-depth insights into organizations' motivations for deploying chatbots for customer service, success criteria for such chatbots and most importantly, organizational factors affecting successful implementation.

Our findings provide new insight into the organizational aspects of chatbot implementation, leveraging the theoretical foundation from general innovation research to contribute to the interdisciplinary body of chatbot research and the emerging research on chatbots within service industries. The findings may also help organizations in laying better groundwork before committing to chatbot for customer service and avoid common pitfalls in chatbot implementation.

The remainder of the paper is structured as follows. First, we present relevant background and, on this basis, explicate the research questions. We then describe the research method and detail the findings on the factors impacting successful chatbot implementation, in addition to the motivations behind chatbot deployment and key success criteria. Finally, we discuss the findings in relation to the presented background and theoretical backdrop, summarize the theoretical and practical implications, and recommend potential directions for future research.

Background

In this background section, we first provide an overview of the application area –chatbots for customer service. Next, since chatbot implementation is a form of innovation implementation, we summarize background on innovation implementation failures. Finally, we review relevant knowledge regarding organizational implementation of technology.

Chatbots for customer service

Chatbots for customer service potentially hold substantial benefits for customer service operations. Gartner (2019) found that about 30% of surveyed companies had already implemented or have near future plans for implementing chatbots. Similarly, a CapGemini report (Taylor et al. 2019) found that in the retail banking and insurance sector, nearly half of top 100 companies have taken up chatbots. In Norway, where this study is conducted, many of the country’s larger corporations and government organizations have implemented chatbots over the last few years as part of their ongoing digitalization strategies (Følstad and Skjuve 2019). The country also sees an increase in the number of domestic chatbot platform vendors a result of this trend.

Typical benefits of a successful chatbot implementation include improvement in operational efficiency (Gartner 2019) and improved customer service experience through strengthened information access and self-service with a personal touch (PwC 2018). Likewise, customers have been found to appreciate the immediacy and accessibility of support provided through chatbots for customer service (Drift 2018). The main motivations for user's uptake of chatbots in general are increased productivity and convenience (Brandtzaeg and Følstad 2017).

A well-designed and implemented chatbot is expected to enrich customer experience and optimize internal operations (PSFK 2018). An inspiring example is the Norwegian telecom provider Telenor's chatbot Telmi, which, in addition to being able to respond to several thousand user intents, also provides support for transactions such service bookings and information about the customer's own subscription (Kvale et al. 2020). Such enriched customer experience through chatbots may create a more engaging brand encounter (Chung et al. 2018) and strengthen positive brand perceptions (Zarouali et al. 2018) and Chatbots and conversational computing has been an area of research for decades. Already in the sixties, Weizenbaum (1966) presented a computer program that could mimic human conversational interaction. This “first chatbot” ELIZA has inspired researchers and practitioners to create machines that can simulate human conversation and, ultimately, reason and present knowledge in a human-like manner (Dale 2016). While chatbots has been explored for customer service purposes since the turn of the century (Lester, Branting, and Mott 2004), recent years' progress in AI and natural language processing, as well as increased uptake of messaging platforms, have spurred a surge of research and industry interest in chatbots (Følstad & Brandtzaeg, 2017).

Current chatbots for customer service are based on advanced technology support for natural language processing (Kvale et al. 2020). Users typically enter their requests in everyday language and the chatbot applies underlying machine learning models to determine the users' intent. Intent prediction, in turn, triggers the associated response in the chatbot. While chatbot content typically is textual or verbal, chatbots may also provide frequently sought options as buttons or quick-replies, as well as content in the form of images and videos, or links to other online sources (Shevat 2017). Keys to successful chatbot implementations include good exception handling routines, capabilities for context understanding to ensure a coherent conversation, facilities for analytics, and integration with backend systems for the chatbot to enable the chatbot to execute certain tasks on behalf the customer (Gartner 2019).

To support simplified initiation of chatbot implementation, many vendors offer "no-code" chatbot platforms, that is, platforms for chatbot implementation requiring little or no software development skills for chatbot configuration and conversation design (Kvale et al. 2019). In practice, this means that once integrated into an organization’s IT system, the importance of the technological elements of the implementation is de-emphasized and, in most cases, the quality of the conservations fed into the chatbots largely determine the quality of the customer interactions. The content and response quality in customer service chatbots are continuously improved by so-called AI-trainers who analyze customer interactions, manage training data, and rework conversational content (Kvale et al. 2019). Chatbots for customer service are also often linked to companies' manned customer service chat, so that requests that cannot be answered by the chatbot may be escalated to a human operator (Følstad and Skjuve 2019). The evolving role of customer service to incorporate an AI-training function and bot-human handoff is typically new to organizations and may require and lead to a range of organizational changes.

Innovation implementation failures

Innovation initiatives are associated with risk of failure. This is also the case for AI-powered chatbots. Gartner (2019) predicted that while there are widespread optimism concerning chatbots for customer service, a substantial part of older chatbot implementations will be discontinued. Furthermore, it is noted that while chatbots for customer service are becoming more common, there is a lag in customer uptake (Forrester 2018). Consequently, substantial efforts have been made to strengthen uptake through enhancing customers' chatbot experience (Kvale et al. 2019; Sands et al. 2020).

The implementation of an innovation entails the transition period from the moment an organization decides to adopt an innovation until the organization's members become skillful, consistent, and committed in its use (Klein, Conn, and Sorra 2001). Klein, Conn, and Sorra (2001) further posit that implementation effectiveness, “the consistency and quality of targeted organizational members’ use of an innovative technology” (p.812), is critical for realizing the intended benefits of an innovation. Reaping the benefits of novel technology for organizations, hence, typically is not a technological challenge but more of an organizational implementation challenge.

Poor effectiveness in innovation implementation is not uncommon and is often observable in the form of failure to achieve the desirable outcomes or failure to deliver results within designated time frame or budget. A review of the information systems literature on IT project failure (Hughes, Rana, and Simintiras 2017) found several key factors contributing to project failure, including poor change management and user resistance, poor project and -requirements management, lack of executive support, and projects being too large and complex. Concerning the implementation of AI systems, a recent survey by the International Data Corporation (Jyoti 2019) found that approximately 25% of the surveyed companies encountered up to 50% failure rate in their AI adoption efforts. The lack of skilled staff and unrealistic expectations were identified as the top reasons for failures.

A five-year multistage study by Sanders and Wood (2020) revealed that many organizations continue to perceive AI-related technologies as “plug-and-play” solutions. Their findings indicate that key to thrive in the age of AI is the development of organizational structures and business models that allow new technology to bring out the best in people. Instead of competing solely on technology, businesses need to focus on human-centric organization models to deliver sustainable competitive advantage. Wilson and Daugherty (2018) argued that service providers need to find new ways of humans and machines to work together, rather than having machines replace skilled human personnel. Jöhnk, Weißert, and Wyrtki (2020) accentuated the importance of organizational AI readiness, characterized by strategical alignment, resource availability, knowledge, culture, and access to data; the first four of these characteristics being particularly relevant to the present study.

Implementation failures of chatbots for customer service may be costly for organizations. A recent interview study of users of chatbots for customer service (Castillo, Canhoto, and Said 2020) found that chatbot interactions may potentially fail, leading to customer anger, confusion, and dissatisfaction. Furthermore, in cases of chatbot failure, users were found to typically blame the chatbot and its host organization, rather than themselves.

Organizational factors impacting implementation effectiveness

While there is a scarcity of research on the implementation effectiveness of chatbots, the broader literature on technology innovation and implementation is substantial and with deep roots. In a recent survey of the literature on service innovation implementation, Singh, Akbani, and Dhir (2020) summarized the factors seen as being particularly relevant to successful implementation of innovations, including organizational structure, leadership, management support, organizational climate, and practices concerning knowledge management and communication. We detail these below.

When discussing organizational structure, Singh, Akbani, and Dhir (2020) particularly addressed the pros and cons of decentralized and bureaucratic job structures – where the decentralized structures with empowered employees often are found to be positively related to proactiveness, whereas centralized authority and regulation can reduce ambiguity. A balanced approach, taking into account organizational factors in technology innovation, is found within theories on sociotechnical systems and human factors. For example, Jones and Smith (2001) proposed that the successfulness of new technology implementation may rely on the “balance” between technology, organization, task, environment and individual. Organizational structures also vary in the degree to which these facilitate employee involvement, participation in decision-making, and the strengthening of employee responsibility through knowledge and information sharing (Hussain et al. 2018). Organizations that promote involvement are often characterized by high-levels of perceived empowerment, information sharing, provision of necessary training and performance-based rewards (Lawler 1986), which may be beneficial for acceptance of technology change (Schraeder, Swamidass, and Morrison 2006).

Leadership and management support are seen as critical for implementation effectiveness in technology innovation and to establishing an organizational climate conducive to innovation (Singh, Akbani, and Dhir 2020). Different styles of leadership may be required at different stages of the innovation process. Oke, Munshi, and Walumbwa (2009) posited that transformational leadership characterized by idealized influence, inspirational motivation, intellectual stimulation, and individualized consideration may be better suited for the exploratory stage of an innovation process. On the other hand, transactional leadership may better for supporting innovation implementation with its concern for management, clear structures, and formal systems, rewards and discipline. Innovation implementation behavior has also been found to improve when management induce trust and affective commitment to change (Michaelis, Stegmaier, and Sonntag 2009). A passive approach to leadership, waiting for employees to gradually adapt to changes, has been found counterproductive and may lead to poor implementation outcomes (Legris and Collerette, 2006 ).

Leaders may influence implementation of innovations by fostering an organizational culture that encourages learning and development, participative decision-making, power sharing, support and collaboration, and tolerance for risk and conflicts (Ke and Wei 2008). In the age of digitalization, the lack of a “digital culture” that is congruent and supportive of the ongoing digital transformation of may prevent an organization from realizing the expected business benefits of digital technology (Wokurka et al. 2017). Digital culture can be defined as “a set of shared assumptions and understanding about organization functioning in a digital context” (Martínez-Caro, Cegarra-Navarro, and Alfonso-Ruiz 2020). Such culture will ideally facilitate collaborative work environments, creativity and innovation, challenges and initiatives, and permanent improvement through a shared digital strategy.

The importance of knowledge diversity and knowledge sharing to implementation success has been much studied (Singh, Akbani, and Dhir 2020). The significance of knowledge management has, for example, been showcased in the uptake and use of platforms for sharing of knowledge and insight for purposes of open innovation (Natalicchio et al. 2017). In a study of implementation effectiveness in small and medium size organizations, Sawang and Unsworth (2011) found the availability of skilled employees to be of significant benefit to innovation effectiveness. Furthermore, the meeting of employees and others representing different backgrounds, disciplines or knowledge bases has been found conducive to innovation success, as for example shown in the positive effects of employee diversity (Østergaard et al. 2011). To identify needed knowledge and know-how, as well as needed human or technical resources, organizations often seek outside help. Lokuge et al. (2019) found that organizational partnerships are important to support technology innovation, and that organizations actively seek and maintain partnerships with vendors and consultants for this purpose.

As a final point, the importance of performance goals and rewards for innovation implementation has been addressed in previous literature – particularly in the literature on innovation implementation. With basis in the work of Klein, Conn, and Sorra (2001), Sawang and Unsworth (2011) distinguished between implementation effectiveness and innovation effectiveness; the former concerning the simplicity or smoothness of the process as well as absence of problems, the latter concerning the benefit of the innovation to aspects such as organizational finances, customer issues, employee factors and even quality of life. That is, relevant performance goals may address the process leading up to a full-fledged implementation of a digital innovation, where efficiency and problem prevention are keys. Alternately, such goals may address the outcome of the innovation process for which a far broader range of performance goals are possible. On the background of this breadth in possible performance goals, it is understandable that Singh, Akbani, and Dhir (2020) note a current lack of broadly accepted measurements for service innovation implementation.

Research questions

While there is a lack of research addressing the organizational aspects of chatbots for customer service (Syvänen and Valentini 2020), the broader research literature on innovation implementation suggests that such digital innovation efforts indeed entail substantial risk. However, this body of literature also indicates a range of organizational factors that are of importance for successful implementation.

This study set out to bridge the current gap in knowledge by investigating some of the impending uncertainties surrounding the implementation of chatbot for customer service. Specifically, to provide a counterweight to the current chatbot literature mainly addressing technical aspects of chatbot innovation or micro-level interactions with users, we address the meso-level organizational factors and implications.

Reflecting the current lack of knowledge, our investigation was guided by open research questions conducive to exploratory research:

• RQ1: What motivates organizations to deploy customer service chatbots?

• RQ2: How do organizations define successful implementation of chatbot?

• RQ3: Which organizational factors affect the successfulness of chatbot implementation?

Method

Research design

An exploratory research approach was selected and was set up as an interview study to gather rich insight into the organizational aspects and implications of customer service chatbot implementation. The study involved personnel from six organizations that had implemented such chatbots and was designed to address three main areas of interest: (a) motivations for implementing customer service chatbots, (b) key success criteria for the implementation, and (c) organizational factors that may impact the outcome of the implementation.

Participants and recruitment

The study used purposive sampling to provide rich insights from a relatively small sample size. The main sampling criteria used to identify relevant organizations were: (1) customer service was an essential part of the organization's service offerings and (2) the organization had experiences with applying chatbots for customer service. Additional criteria were used during sampling to ensure variability in terms of (1) the duration of the organizations' chatbot usage, (2) business sector, (3) and chatbot platform provider used by the organizations.

Within organizations fitting our sampling criteria, we invited participants holding different roles related to the customer service chatbot and its implementation/maintenance, including project managers and members, product owners, customer service, and AI-trainers.

Six large organizations (number of employees >1,000) were involved in the study. These organizations represented various lines of businesses, had up to 2.5 years experiences in using chatbots for customer service, and used the platform of one of the top three chatbot platform vendors in Norway. From these organizations, a total of 14 participants were recruited for semi-structured interviews. Table 1 summarizes the business sector of the participating organizations, the number of participants from each organization (coded as Organization A–F) as well as the participants' roles, and the organizations' choice of platform provider (coded as Vendor A, B, and C).

Table 1. Details of participants and participating organizations.

Organization code	Business sector	Number of participants	Participant role	Platform provider (vendor)
A	Bank and finance	4	Project lead, product owner, head of customer service, product owner, customer service advisor/ AI trainer	A
B	Media and telecom	2	Product owner, AI trainer	A
C	Public sector	3	Project lead, project team member, head of customer service,	A
D	Bank and finance	1	Technological strategist	A
E	Consultancy	2	Product owner, middle office/AI trainer	B
F	Media and telecom	2	Product owner, head of customer service	C

Interview guide and data collection

The interview guide was developed on the basis of existing literature on technology innovation implementation and change management, as well as pilot interviews with subject matter experts. The guide covered six main themes: (1) background of company and participant (the overall process and experience of chatbot implementation in the organization, as well as the participant’s role); (2) the organization's motivation for introducing a chatbot and how “successful chatbot implementation” is defined; (3) organizational factors that affect the chatbot implementation process and its outcomes; (4) the impact of chatbot implementation on the organization and work processes; (5) the impact of chatbot implementation on customer service; (6) key learning points from the organization's experience and future direction of chatbot for customer service.

The interviews were conducted in the period August–October 2019. As the study involved organizations in Norway, all interviews but one were conducted in the Norwegian language. All participants were interviewed individually. Twelve interviews were conducted face-to-face at the participants’ business premise, and two were conducted virtually. The duration of the interviews was 45 − 80 minutes. All interviews were audio recorded following participant consent. The participants received no compensation for their participation.

Data analysis

Audio data from the interviews were transcribed using non-verbatim transcription. The codebook approach (Crabtree and Miller 1992) of thematic analysis, specifically Template Analysis (Brooks et al. 2015), was chosen for data analysis as it was deemed adequate and feasible for the purpose of this study.

The analysis was carried out iteratively, following the six-step procedure outlined by Brooks et al. (2015). A total of 82 preliminary codes and 12 a priori themes were generated from initial coding and further refined into an initial coding template, which was then adjusted and refined to ensure that the final template could capture the richness of the entire data set. Table 2 shows the final template used to code and analyze the data.

Table 2. Final coding template in the thematic analysis.

Themes	Second level themes	Lower-level themes
1. Participant's role and background of implementation	1.1 Participant's role in organization and project
	1.2 Chatbot implementation process
	1.3 Chatbot role in customer service	1.3.1 Complementary chat channel 1.3.2 Gatekeeper to chat channel
2. Motivations	2.1 Address high-volume repetitive questions
	2.2 Improve customer service experience
	2.3 Reduce cost and resource requirements
	2.4 Drive digitalization
3. Success criteria	3.1 Successful issue resolution
	3.2 Traffic improvements
	3.3 Feedback from customers and third parties
4. Organizational factors	4.1 Work and team organization
	4.2 Change management	4.2.1 Resistance management - employee engagement and sense of ownership 4.2.2 Leader's roles (1. granting team autonomy, 2. attitude toward innovation) 4.2.3 Organizational dimensions (1. lengthy decision process and complex organization, 2. regulatory and security concerns, 3. information to and from stakeholders)
	4.3 Competency management	4.3.1 Required experience and skill 4.3.2 Primary learning mechanism: learning by doing, workshops by vendors 4.3.3 Secondary learning mechanism: experience sharing in teams and organization
	Organizational resources	4.4.1 Human resources 4.4.2 Technology resources (1. technology readiness, 2. platform and vendor partially determine in-house resource requirements)
	Performance measures	4.5.1 Qualitative and quantitative measures 4.5.2 Challenges and implications of existing performance measures
5. Learning points	5.1. The importance of understanding the technology
	5.2 Continued need for customer service personnel
	5.3 No one-size-fits-all to chatbot implementation	5.3.1 Meticulous planning vs. trial and error 5.3.2 Single vs. multiple chatbots

Method quality and ethical considerations

The research approach and method were designed to fulfill commonly used quality assessment criteria for qualitative research: transparency, reflexivity, and transferability (Treharne and Riggs 2015). Transparency is addressed through conducting the analysis in a clearly demarcated stepwise approach. In the presented study this is supported by the inclusion of coding examples, initial and final template, as well as citations of participants’ response where appropriate. The second criteria, reflexivity, concerns the researchers' efforts throughout the research process to reflect upon and “explore the ways in which a researcher’s involvement with a particular study influences, acts upon and informs such research” (Nightingale and Cromby 1999, p. 228). In this respect, we prudently reflected on the first author’s prior working experience in customer service and the second author’s ongoing active engagement in chatbot-related studies, and how this could influence data collection, data analysis, interpretation and presentation of findings. The issue of transferability was addressed through the sampling criteria used in this research. Despite the relatively small sample size, transferability of the findings was assured through the breadth of businesses sectors and chatbot vendors represented by the participating organizations, as well as the participants’ varied roles in the organizations.

All relevant ethical standards pertaining to qualitative interviews were adhered throughout the study. Participation was fully voluntary, informed consent was acquired prior to interviews, and participants were informed that they could withdraw at any point in the study. The privacy aspects of the study were assessed and approved by the appropriate regulatory body. All data were anonymized following transcription. Measures to ensure that the participating organizations are not identifiable in the final dataset and publication were also taken, for example, some of the quotes found in the Results section have been paraphrased to avoid possible identification of companies.

Results

In the results section, we first provide an overview of our findings concerning the participating organizations' chatbot implementations, their motivations for implementing customer service chatbots and the criteria by which the success of the chatbot implementation were assessed. Following this, we present the findings on the organizational factors identified as the key to successful implementation. Finally, we briefly summarize what our participants had learnt from their experience in chatbot implementation.

Implementations, motivations, and success criteria

There were slight variations in the way the participating organizations had implemented their chatbots. Some organizations had carried out extensive pilot studies, while others had spent less time and resources on this. All organizations had engaged a third-party chatbot vendor; four used chatbot Vendor A, the other two organizations used Vendor B and C respectively. Three of the organizations, had the chatbot implemented as a gatekeeper, that is, a first point of contact for customers making requests through the chat channel. In these three implementations, customer would be directed to a human operator only if the chatbot failed to provide a satisfactory solution. The other three organizations had implemented the chatbot as a complimentary channel for customer service and retained the option to chat directly with human customer service personnel during operating hours.

With these variations between the participating organizations in mind, we identified four main motivations for implementing a customer service chatbot. First, chatbots were reportedly seen as potential solutions to solve high volume and repetitive questions. Second, the participants reported on expectations concerning improvements in customer service experience; particularly as a consequence of the chatbot's ability to provide immediate support around the clock. Third, a few participants mentioned the pressure to reduce cost and resource requirements as one of the key motivations for implementing a customer service chatbot. Finally, nearly all participants reported on a sense of need within the organization to digitalize service offerings and to add value to the existing customer service in light of changes in the market and increased competition. This latter sentiment is exemplified in the following quote:

In the beginning, it was like “We need to be part of this, we cannot be late in the market. We need to make sure that we are ahead of development.” (Participant 11)

To keep track of their chatbot implementation, all participating organizations had applied several evaluation criteria against which to assess implementation success. Three common criteria for defining a successful chatbot implementation emerged from our analysis of the interview data. First, successful issue resolution; nearly all participants stated successful issue resolution as the main criterion used in their evaluation. Second, traffic improvements; the majority of participants also indicated that they were considering the reduction in customer service traffic and expansion in service capacity as an important deliverable. Third, positive feedback; some organizations relied on customer feedback and reviews by third party to obtain more in-depth information about their customer experience in relation to the chatbot, identify potential area of improvements, and find out how they fare in comparison with their competitors. As noted in the following participant quote:

Primarily, that the customers get faster answers. The queue goes down […] but these change continuously, so it is difficult to look at these as measures of success. We also need to look at the dialogues. (Participant 1)

Some participants offered more tangible definitions of “successful implementation of chatbot”, for instance, that the chatbot was considered successful when it achieved a specific score in the customer satisfaction rating. Other participants reported greater fluidity in the organization’s definition of successful chatbot implementation, as in the following quote:

We had a target of having [fallback rate] at below a certain percentage initially, that it should be able to answer nearly all requests. […] It had hit the target, but then it became lower again, and now we are somewhat there. (Participant 8 – paraphrased)

There is some overlap in the motivations of implementation and evaluation criteria identified through this research. For instance, the length of waiting time in customer service queue and the ability of the chatbot to provide satisfactory solution inevitably affect customer service experience. Despite the variation in how organizations define successful implementation of chatbot, all participants had nonetheless acknowledged that certain organizational factors must be in place to ensure successful implementation of chatbot and post-implementation performance.

Organizational factors in chatbot implementation

The primary focus of this study is the investigation of organizational factors that may influence the outcomes of implementing a customer service chatbot. Based on the interviews, five such factors were identified: work and team organization, change management, competencies and competency acquisition, organizational resources and performance measurement. In this section, we detail our findings on these factors.

Work and team organization

All participants reported the importance of work and team organization on the process of chatbot implementation. Specifically, the organization of work and personnel for chatbot training and content management – the AI-trainers. The skills and knowledge of the AI-trainers represent a new set of competencies to be developed.

Due to the importance of providing good and updated chatbot content, and training the chatbot in response to customer input, nearly all participants emphasized the importance of competent and dedicated AI-trainers in the implementation of chatbots for customer service. However, although AI-trainers are a new category of employees, all organizations participating in this study reportedly recruited AI trainers internally from their existing pool of customer service personnel rather than hiring new personnel for this purpose. Hence, setting up a chatbot team required not only the establishing of novel work processes and team structure, but also the facilitation and management of the evolving role of customer service personnel to serve as AI-trainers. As noted by one of the participants:

It’s about bridging that understanding of the problem, where we want to go, and the different problems that need to be addressed to move in that direction, and working it out in a good way. So with good people, it should happen, but you need to empower them to make it happen. (Participant 7)

In consequence, while not all participants reported substantial changes to their own workflows upon implementation of their customer service chatbot, most participants reported the need for a change in work organization throughout the different phases of chatbot implementation. This change in work organization was typically organized as a time division for the involved personnel. For all but one of the organizations, the team of AI-trainers consisted of personnel working part-time on AI-training and part-time on their regular customer service tasks. Most participants argued that such split arrangement allowed AI-trainers to better maintain their customer service skills and stay updated of changes in service offerings, available resources, and customer requests.

For some, a split arrangement between AI-training and regular customer service tasks was associated with periods of increased workload, as both sets of tasks tapped into the same resource pool. This increase in workload due to AI-training was reportedly more noticeable in organizations with relatively small customer service departments. Although this increase in workload was partially offset as the introduction of chatbot removed some of the inquiries from manual queue, the smaller departments still experienced more strain during the initial phases of AI-training. This is illustrated in the following quote:

We have a small customer service department, when two of us start to work on AI training, the impact is much larger than say, if you compare it to our parent company with a much larger customer service department. (Participant 1)

Task assignment to the AI-trainer role was typically reported as flexible. Many participants generally welcomed this flexibility. However, it was also reported that the lack of clearly defined roles, responsibilities and procedures could slow the implementation and maintenance of the chatbot, especially in the case of cross-functional teams or teams with members situated at different locations. In consequence, some of the participating organizations were reported to gradually prioritize the refining of routines and structuring of collaboration, role definition and governance. As noted in the following quote:

If you do not have clear governance, clearly defined roles and responsibilities, then you are failing on the front, then it will be really difficult to do it well. (Participant 7)

As an additional challenge to the novelty of the AI-trainer role and the organization of work and organization of chatbot implementation and maintenance, the participants also noted the need to carefully consider the transition from the chatbot development project to its operational state. Several participants noted disagreements during this shift, such as which department should take over, long-term resource commitment and modifications to existing work procedures, and called for careful consideration of transition and succession plans to mitigate these challenges.

Change management

Most participants noted change management as an important factor in customer service chatbot implementation, which concerns efficient management of both desirable and less desirable changes. Based on the participant reports, four elements were detailed: leadership, resistance management, decision-making processes, and information management.

In terms of leadership, employee autonomy and managerial support of the chatbot development project were noted as highly important to successful implementation. According to some participants, the leader’s attitude toward digitalization could affect the level of support they received in resource acquisition.

We have had supportive managers throughout, and also had a department that encourages innovation […] We have been allowed to find our way (Participant 9)

Resistance management was also noted as important. Although most participants did not perceive substantial resistance in the organization with regard to the chatbot implementation, two primary sources of resistance were reported: skepticism toward the ability of chatbot to deliver high quality customer service and worries related to job security. In particular, employees who were not part of the implementation project might perceive the chatbot as an inadequate tool for customer support or as a threat to their job as customer service personnel.

Most of the participants considered active employee involvement as an effective way to alleviate these concerns. To strengthen involvement and further chatbot improvement, AI trainers actively attempted to engage coworkers outside of the chatbot project by consulting broadly on matters such as semantic choice or the appropriate answers for questions not within their area of expertise.

We are good at involving others, we really are. This is important. […] The customer service personnel know so well what works and what does not. (Participant 3)

Information management was also noted as an important aspect of change management. While nearly all participants stated that their organization had disseminated information on the chatbot within the organization through multiple formal and informal channels, some noted that they were often presented a “glorified story” that understated the potential challenges that come with chatbot implementation. In addition, a few participants also reported a lack of information on plans for employees whose job would eventually be affected by increased level of customer service automation.

On several occasions when we were discussing about backend integration of our virtual agent, they said, “Oh well, it’s going to take our job”. It was kind of a joke, but still, people get that this is a part of development and it will reduce the need for customer service workers. (Participant 10)

Almost all participants reported that the lengthy and complex decision-making processes in their organizations had also somewhat hampered the progress in chatbot implementation. In addition, for chatbot teams without cross-functional team members, the participants also indicated that the reliance on other departments such as IT or sales department could slow the implementation progress.

Many participants acknowledged that there was still a lack of structured flow of information to and from various stakeholders beyond the general information about chatbot disseminated through company intranet. While some participants had reportedly taken a more proactive stance to facilitate information exchange, others remained somewhat reactive. As chatbot’s ability to provide accurate answer is partly dependent on the chatbot team’s continuous efforts to keep the contents updated, some participants were concerned that the lack of coordinated information flow could negatively affect customer experience.

We have a communication and training responsible person. […] If there is a new product, for example, this person gets the information. […] And when we make changes in the chatbot, we should inform those that will be affected by it. (Participant 11)

Competency management

Having the right competencies and competency acquisition mechanism within the chatbot team was reported by the participants as another critical factor in ensuring successful implementation of customer service chatbot. While chatbot platforms differ in the functionalities they offer, the participants noted that the main distinguishing characteristic between chatbot implementations, and hence, the chatbot’s subsequent capacity to serve as a good customer service tool, is largely determined by the content and training provided by the chatbot team, specifically the inputs of AI-trainers.

Three core skills were identified by participants from all organizations as prerequisites to the role of AI-trainers: (a) prior experience with customer service, (b) good writing skills and (c) analytical abilities. Internal recruitment was the preferred method of talent acquisition for all participating organizations, as nearly all product owners or team leaders of the chatbot teams found that context-specific customer service skills and familiarity with the products/services offered by the organization, as well as general knowledge of how things work in the organization to be key determinants of AI-trainers’ ability to realize the full potential of the chatbot.

[AI-trainers] must have knowledge from the customer center and very good knowledge in the product, as well as the complexity of how things are done, and you need an interest in technology (Participant 13)

Diversity in product knowledge was also valued by most of the participants, and although interest in technology was appreciated, little or no emphasis was placed on technical competency. Most organizations reportedly did not offer formal training for their chatbot team. Instead, employees enrolled in workshops and certification courses run by the platform vendor and subsequently enhanced their skills through “learning by doing”.

Training should include information on everything, the technical aspects, what happens from the moment customer send us an inquiry and it goes through APIs, which then activate the classification algorithm that places the customer inquiry at the right intent, and subsequently provide chatbot with the answer to the inquiry. (Participant 12)

Experience-sharing between team members, or even across subsidiaries, was also noted by many participants as a valuable means of competency acquisition. However, most participants acknowledged that their organizations had yet to prioritize the establishing of routines or procedures for this type of learning, and most exchanges occurred impromptu.

Organizational resources

Most participants stressed the availability of organizational resources as key to successful chatbot development. Such resources include the readiness of the existing system infrastructure to enable smooth transition to automated chat, as well as the availability of competent AI trainers and in-house IT developers. Furthermore, since all participating organizations had acquired their chatbot through third party vendors, the participants also considered the chatbot platform and its provider as a valuable resource in the implementation of chatbot. Most of the participants indicated that they regarded their platform provider as a collaborative partner who provided not only the platform and its relevant training, but also continuous supports and mutual learning opportunities that enabled continuous improvement and growth on both sides.

[The platform vendors] have been really good at following up. […] We want a best possible use of the platform, so this has been an important part of the implementation. (Participant 8)

Nearly all participants conceded that they had underestimated the resource requirement for chatbot implementation. Many participants acknowledged that they had anticipated minimum resource requirement partly due to their misconception of the AI technology, whereby the notion of “intelligent’ robot” led many to believe that chatbot could minimize the need for manual customer services almost instantly.

“The hype around the chatbot or virtual agent is a problem because it is inhibiting, it makes people underestimate the seriousness and the investment needed to make it fly, both in terms of the people who have to create the conversations, ensure that the understandings and stuffs are adequate behind and of quality.” (Participant 7)

It is noteworthy that resource requirements were reported to fluctuate significantly during the different phases of the chatbot implementation, as it progressed from introductory to operational phases. Such fluctuation was seen as challenging to accurate projection of resource requirement. Hence, many participants noted that they would like to see more flexibility in resource allocation and continuous resource requirements for subsequent improvement works.

Performance measures

A final factor noted by many participants, was the availability of good performance measures for the customer service chatbot. Such measures might improve the appeal of the business case for chatbot implementation, which in turn could help to secure the resources required for subsequent improvement works. However, from the reports of the participants, such performance measures appeared to be less than straightforward for chatbot implementation.

All participants indicated that the organization had used multiple types of performance measures, and at the same time, they still noted disagreements and uncertainties regarding the chatbot performance.

Measurements are important, but it is also important how these are used. We cannot just measure the bot on any numbers, they need to actually measure performance (Participant 9)

Both quantitative and qualitative measures were used by all participating organizations. Some of the quantitative measures named by participants included statistics of incoming customer traffic through various service channels (typically telephone, chat, email or physical service counter), customer satisfaction ratings following chats, and the number of customer inquiries received and completed by customer service personnel. Qualitative measures used included analysis of chatbot dialogues as well as customer feedback gathered at the end of chat conversation or through other more elaborate customer survey.

Many participants felt doubtful about the ability of certain quantitative performance measures to accurately capture the effects of chatbot implementation on organizational and employee performance. For instance, while all organizations actively monitor the fluctuation in telephone traffic, several participants noted that a change in incoming telephone traffic could never be fully indicative of the chatbot’s performance, as factors such as random and seasonal fluctuation in telephone traffic, increase in self-help functions on the company website and the shift in customer preference toward digital means of communication could also affect telephone traffic.

Similarly, some participants observed instances where customers gave low ratings or unfavorable feedback to the chatbot due to reasons not attributable to the chatbot, for instance, to convey annoyance caused by unfavorable (but correct) answers or personal dislike for automated services. Responder anonymity was also found to challenge the benefit of performance measures, as the organizations did not have the opportunity to follow-up much of the received feedback from customers, hence not being able to gain in-depth insight into causes of (dis) satisfaction.

The participants' reflections on performance measures also concerned an interesting lack of consensus regarding what constituted adequate key performance indicators (KPI). While some participants argued that a chatbot for customer service should be assessed using similar KPIs as for customer service personnel, others argued that the chatbot should be assessed by KPIs for technology support. Participants who posited that customer service KPIs were also applicable to the chatbot argued that since it was a customer service chatbot, its performance should be measured in the same way as a service personnel. On the other hand, some participants questioned the benefit of using customer service KPIs to measure chatbot performance. At the time of data collection, none of the participants' organizations had explicitly linked performance rewards to the successfulness of the chatbot implementation.

We have very good performance indicators for our customer representatives […] But we cannot measure chatbot in the same way. It is after all, not a human being. If a chatbot told you “No, I can’t do this”, you will most likely be annoyed and displeased. But a human being could sugarcoat it in a way that will make you accept the answer somehow. (Participant 2)

Main lessons learnt

Toward the end of the interviews, the participants were asked to reflect on the main lessons they learned from chatbot implementation and operation. The lessons learnt may be summed up in three key points: (a) understanding the chatbot technology, (b) acknowledging that chatbots do not eliminate the need for customer service personnel, and (c) the lack of one-size-fits-all solutions.

The right understanding of the chatbot technology was reported as essential to the chatbot implementation. This understanding affects the organization’s ability to create an appropriate scope of the chatbot, and consequently, its ability to estimate accurately resource requirement and appropriate evaluation criteria at various stage of implementation. Specifically, such understanding is important to counter initial conceptions of chatbots as an easy fix to providing efficient customer service. While potentially a valuable supplement to customer service, the participants noted that a chatbot project initiated with unrealistic expectations and a lack of knowledge concerning the state of the art of the technology will likely have a difficult road ahead.

The second lesson conveyed by the participants was that a chatbot does not eliminate the need for customer service personnel. At this point in time, owing to limitations in the technology and the readiness of customers to accept full-scale service automation, manual services are still required in most business sectors. Instead of replacing customer service personnel, the implementation of chatbots has brought about positive changes to the traditional customer service role, serving to expand a traditionally rigid role into one that includes more flexibility and autonomy. Specifically, the increase in analytical tasks was perceived by many participants as an exciting change.

Finally, the participants noted that there was likely no one-size-fits-all solution to how a chatbot implementation should be carried out. Rather, adaptation to the particular organizational context and business sector was considered a more favorable practice. This need for contextual adaptation was also seen in the totality of the findings. Some participants saw less value in spending resources in extensive user testing and pilot study, while others believed that they had benefited greatly from meticulous planning and extensive pilot study. A split-task arrangement between AI-training and customer service that worked well in some organizations might be less effective in other organizations. Furthermore, while some while others preferred to have multiple chatbots to manage the complexity that comes with chatbots covering larger areas of services, others may see having a single chatbot to serve multiple subsidiaries and product lines as a source of synergy – potentially invigorating and strengthening customer service.

When you start with many different technologies, it’s good you can say that’s for experimenting, but you also have got to have an exit plan to consolidate into one at some point in time (Participant 7)

Summary of results

Concluding the results section, we summarize the main results in Table 3 below.

Table 3. Summary of main results for each main theme in the analysis.

Themes	Summary of results
1. Participant's role and background of implementation	Approaches to chatbot implementation vary, in terms of the priority given to piloting and trials during the implementation process as well as the role of the chatbot in customer front end. The chatbot's role may be that of a gatekeeper, as the only access point to chat customer service, or that of a complementary channel to chat customer service.
2. Motivations	Four motivations for chatbot implementation were identified: (a) handle high volume of repetitive questions, (b) improve customer service experience, (c) reduce cost and resource requirements, and (d) drive digitalization. Digitalization is seen as important for added value and increased competetiveness.
3. Success criteria	All participants reported on established success criteria for their organization's chatbot. These included (a) successful resolution of customers requests, (b) improved management of traffic to customer service, and (c) positive feedback from customers and third parties. The concreteness of the success criteria varied substantially.
4. Organizational factors	Five organizational factors of particular relevance to chatbot implementation were identified: (a) work and team organization, in particular how work and teams are organized to establish and maintain chatbot content and training; (b) change management, where supportive leadership allowing for employee autonomy was seen as key to reduce resistance toward change; (c) competency management, important to acquire needed experience and skill, specifically for the new role of AI-trainers, (d) organizational resources, including the readiness of the existing technical platforms, availability of third-party resources, as well as availability of skilled personnel in-house; (e) and performance measures, which were seen as highly important but also challenging as their format and application was not yet firmly established.
5. Learning points	The participants made specific note of the following learning points: (a) the importance of understanding the technology to adequately scope the chatbot and plan for needed resources, (b) the importance of acknowledging that customer service personnel is still needed even with effective chatbots, as they take on different roles, and (c) that chatbots for customer service must be carefully tailored to meet the needs and characteristics of the organization for which it is developed and implemented – there is likely no one-size-fits-all solution.

Discussion

In response to the lack of knowledge regarding organizational factors impacting the implementation of chatbots for customer service, we have presented the findings from a study involving six organizations that have implemented such technology. While our participants noted that there hardly is a one-size-fits-all solution to chatbots for customer service, we nevertheless were able to explore several relevant organizational factors of importance. In the following, we first discuss our findings with regard to the research questions and related works. Thereafter we summarize theoretical and practical implications, remarks on the limitations of this research. and recommend for future research.

Motivations and success criteria for chatbots in customer service

While AI-powered chatbots hold substantial potential for improving customer service in organizations (Gartner 2019; PwC 2018), the implementation of such digital innovations entails substantial risk (Hughes, Rana, and Simintiras 2017; Jyoti 2019). Hence, it is important to understand organizations’ motivations and success criteria for chatbots for customer service, why the risks in implementation are considered outweighed by the benefits and what it takes for companies to achieve a successful chatbot implementation.

Our findings on organizations' motivations to take up chatbots for customer service were in line with findings from current industry reports (Gartner 2019; PwC 2018): automated processing of routine requests, operational efficiency, customer experience, and a drive for digitalization. It is comforting for the future of customer service that the participants were found to put substantial weight on both efficiency in service provision and strengthened customer experience. Interestingly, the motivation to implement chatbots to address high volume repetitive requests may both improve operational efficiency – through reduced need for personnel to attend to these requests – and strengthened customer experience – due to more immediate responses to simple routine requests and better access to skilled personnel for the more challenging requests. It is also worth noting that organizations may be motivated not only by the immediate benefits of taking up chatbots, but that they also see the uptake of novel digital technology as a means toward staying relevant. This latter motivation – a digitalization drive – is interesting, as it may possibly explain some of the observed lack of precise success criteria and performance measures for chatbots for customer service.

The reported success criteria for chatbot implementation closely followed the reported motivations, namely issue resolution (necessary for automated processing for routine requests), traffic improvements (important for operational efficiency), and positive feedback (a key indicator of positive customer experiences). Furthermore, we find the noted diversity among the participating organizations with regard to the precision in established success criteria – where some were highly targeted (e.g., aiming for a specific customer satisfaction rating) and others more loose or fluid – as potentially reflecting variations regarding the participating organizations transitions from an exploratory stage of the innovation process to an implementation stage (Oke, Munshi, and Walumbwa 2009). A more targeted set of success criteria is likely to be expected when the implementation has reached mature stage, and the organization has a clearer view of what to realistically expect from the innovation.

Reviewing the identified motivations and success criteria in light of innovation adoption theory (e.g., Jöhnk, Weißert, and Wyrtki 2020), we find that the innovation's characteristics, organizational characteristics and management's commitment to innovation are of higher relevance in comparison to environmental factors. That is, motivations seem to reflect more of the perceived fit between the innovation and the organization, rather than an analysis of the competitive advantage potentially brought about by introducing chatbots.

Organizational factors impacting chatbot implementation

Our findings provide detail insight into five organizational factors that particularly impact the chatbot implementation: work and team organization, change management, competency management, organizational resources, and performance measures. On a high level, the set of factors closely adhere to what may be expected – given the general literature on service innovation implementation (Singh, Akbani, and Dhir 2020). At the same time, the findings concerning these factors represent valuable insight into the specific characteristics and demands presented by chatbot implementation. Hence, the presented findings serve to extend current knowledge of service innovation to the specific domain of chatbots for customer service.

Work and team organization were found to have substantial impact on the chatbot implementation. The introduction of chatbots for customer service entails the definition of a new employee role – that of the AI-trainers (Kvale et al. 2019), and a new way of organizing work to accommodate this new role. AI-trainers, typically being recruited based on their competency as customer service personnel, as well as their communication and analytical skills, form the basis of the new teams dedicated to develop and maintain the content of the chatbots. However, these teams are often an integral part of the customer service department and AI-trainers typically also use part of their time as customer service personnel. Such work and team arrangements are found to require a certain level of decentralized structure to ensure employee proactiveness in filling the new role (Singh, Akbani, and Dhir 2020), and it will be important to find a balance between the technology demands, the needs of the organizations, and the requirements and motivations of the individual employees (Jones and Smith 2001). In particular, when establishing the chatbot work and team organization in general, and the AI-trainer role in particular, our findings indicate the benefit of high levels of person-job integration, where the team and its individuals are trusted and free to make decisions on how to best implement their work, thereby creating an environment conducive to employee involvement (Hussain et al. 2018).

Constructive change management for chatbot implementation was found to be supportive of establishing needed work and team organization. The value of leadership in facilitating the innovation process while at the same time empowering employees was noted, along with leaders' ability to building and maintaining a culture conducive to digital innovation and to mitigate change resistance. For this purpose, engaging employees and organizational stakeholders beyond the immediate project team was seen as valuable, as the success of the implementation was seen as partly dependent on the buy-in from the entire organization. For this purpose, management support was also seen as of particular importance. As such, the findings resonate well with the existing literature. Where transformational leadership (Oke, Munshi, and Walumbwa 2009) is critical in initial phases, while including measures to engage the organization in the innovation process and facilitate an affective commitment to change (Michaelis, Stegmaier, and Sonntag 2009). It should be noted that while building a culture for digital innovation clearly is important (Ke and Wei 2008), there is also a need to make sure that employees see the particular changes brought about by the chatbot as beneficial, potentially enabling them to provide better customer service, without being a threat to their own job situation. Uncertainty in this regard may potentially be negative to implementation success.

Competency management was found to be of critical importance to the successful implementation of chatbots for customer service. In particular, this was due to the needs arising from the new role of AI-trainers as described above. Implementing chatbots for customer service required the organizations not only to acquire novel technology, but also to build competency to implement and maintain this technology. Echoing the findings of Sanders and Wood (2020) in their study of AI technology implementation, organizations have to acknowledge the need to build needed competency inhouse. Interestingly, despite the novelty of the needed competency, recruitment of personnel typically was done within the organization – drawing on the existing pool of skilled customer service employees – rather than going externally for new personnel. This may, in part, be due to the need for AI-trainers to be knowledgeable of the customer service needs related to products and services of the organization. In addition, the sparseness of skilled AI trainers in the job market due to the novelty of AI-trainer role might have also limited the organizations’ ability to hire externally. It is also noteworthy that the studied organizations were partly relying on their platform vendors to help train and support AI-trainers – reverberating the idea of innovation processes as depending on collaboration between personnel and teams within and outside the organizations (Natalicchio et al. 2017; Østergaard et al., 2011).

Organizational resources of relevance for implementation of chatbots for customer service, as discussed by our participants, in particular included the readiness of existing system infrastructure and the availability of personnel fulfilling the different roles in the implementation process. The participants typically depended on external vendors to fill parts of the resource needs, accentuating the benefit of collaboration with external partners for such innovation work (Lokuge et al. 2019), but also accentuated the need to access or develop needed resources inhouse. A notable finding in relation to resources as a critical organization factor in chatbot implementation is the organizations’ frequent underestimation of the resource requirements for chatbot training, which encompass both technical and non-technical resources. In particular, the part of the implementation process where the quality of chatbot may be highly dependent on the AI-trainers’ ability to dedicate time for the development and maintenance of the chatbot conversational content and training data. Underestimation of resource requirements may in part be due to the organizations’ inexperience in chatbot technology, and in part due to an initial lack of understanding of what the production of chatbot conversational content and data entails. The challenge in estimating the resource demands of chatbot implementation is reminiscent of Sanders and Wood (2020) finding that organizations often consider AI-related technologies as “plug-and-play” solutions, underestimating the manual work required to fit these technologies to the needs of the organization.

Performance measure was the fifth and final organizational factor detailed in our findings. Resonating the breadth of possible performance measures suggested in the literature (e.g., Sawang and Unsworth 2011, we found the participating organizations to apply a range of measures. However, as noted by our participants, the definition and application of the measures could be challenging. An added challenge for the chatbot domain, compared to other digital innovation, was that organizations were conflicted whether the performance of the chatbot should be measures using similar approaches and benchmarks as for human customer service personnel, or whether it should be measures as other automated self-support technologies such as self-service customer websites and apps. Hence, while performance measures clearly are important for innovation efficiency for chatbots for customer service, it is currently undecided which of the available performance measurement candidates one should apply, and which should be the benchmark for comparison.

Implications

The presented findings hold several implications for theory and practice. In the following, we summarize those which we consider to be key.

Implications for theory

Organizational factors impacting chatbot implementation echo factors impacting innovation in general. As such, the presented findings indicate the validity of existing general theory on implementation of digital innovations for the chatbot domain. Nevertheless, our findings also suggest that the domain of chatbots for customer service holds important particularities of theoretical relevance. We consider the following implications for theory as of particular interest:

• The dual aim of operational efficiency and customer experience: In service research, a duality is seen in either providing operational efficiency, seen by customers as efficient problem resolution, or excelling in customer experience, by surpassing customer expectations (Dixon, Freeman, and Toman 2010; Følstad and Kvale 2016). However, innovation in the domain of chatbots for customer service allows organizations to nurture both these aims. As such, the findings may prompt a rethinking of previous duality between efficiency in operations and superiority in customer experience.

• The repurposing of personnel in response to AI-driven innovation: Within social science research, there has been substantial interest in the implications of the so called “AI technology revolution” on the job market (Frey and Osborne 2017). In this context, the emerging role of AI-trainers is a highly interesting case of repurposing of personnel in response to automation through AI and, as such, of substantial theoretical interest. The case is particularly interesting as AI-trainers are recruited from inhouse customer service departments and that they benefit from maintaining the links with these departments even after role change.

• The competency required by emerging AI innovations: Innovations based on AI technologies may be viewed as either competence-destroying or competence-enhancing (Paschen, Pitt, and Kietzmann 2020). Furthermore, it is commonly acknowledged that AI-based innovations will require organizations to acquire new competencies. Our study serves to showcase an additional dimension to this, that is, the need for existing competency to develop and refine current AI-based innovations. While this was taken for granted in previous symbolic AI, in work on expert systems, current AI solutions are typically seen as data driven (McAfee & Brynjolfsson 2017 ). However, even for the AI innovations of today, the competency needed to benefit from these may, at least for some domains, be developed from existing competencies.

Implications for practice

The presented findings also hold useful implications for practice. Specifically, we believe that organizations aiming to implement chatbots for customer service will find the following implications to be of relevance to improve implementation efficiency and avoid pitfalls.

• Support employee empowerment: Our findings suggest the benefits of allowing substantial autonomy and empowerment of the team and individuals in charge of the chatbot implementation. This is in part due to the innovation necessitates building of a team structure and work processes for which the organization may not have an existing blueprint for. Furthermore, the evolving character of the chatbot technology, where organizations only partially understand its opportunities and limitations, may require an empowered team and individual employees with sufficient autonomy, motivation and insight to make the needed adjustments as they progress. The benefit of empowerment and autonomy may in particular be valuable in initial phases of implementation.

• Clarify resource demands: The organizational resource demands of chatbot implementation was typically reported to be underestimated up front, in part due to a limited understanding of the chatbot technology and the need for extensive content development and training of the chatbot. To support informed budgeting and prioritizing of resources, it will likely be valuable for organizations to seek information on organizational resource requirements for this non-technical part of the innovation implementation project.

• Mind the AI-trainer: The introduction of chatbots for customer service entails the introduction of a new role with a new set of competencies. To be successful in implementation, organizations need to prioritize the acquisition of AI-trainers – likely by building this competency inhouse through a collaborative partnership with the chatbot platform provider.

• Refine performance measures: A range of measures for the chatbot performance was reported by our participants. At the same time, establishing a coherent set of measures that satisfy needs for monitoring and improving performance may be seen as challenging. Refining available performance measures into a coherent toolset for monitoring and improvement is recommended.

Limitations and future work

While this study contributes initial insight into the organizational aspects of chatbot implementation, it also has important limitations. Specifically, the study is relatively small size and limited to a particular region. Furthermore, it does not include multiple sources of data collection, and it is conducted at a single point in time. In the following, we discuss these and suggest future work motivated by the limitations in what has been presented.

The size and regional coverage of the study represent a potential limitation in the generality of the study findings. Given the exploratory aim of the study, where the aim is to provide initial insight concerning key organizational factors impacting chatbot implementation, we do not see this as a critical limitation. Nevertheless, it would be beneficial with studies validating and extending our findings involving a larger number of organizations with cross-regional coverage.

The study is also limited in being an interview study, as it does not take into account other data sources – such as document studies, observational studies within the organization, or studies of the chatbots during or after implementation. Hence, we cannot exclude a bias in the findings due to the self-reported character of the data, though we have no reason to suspect such bias. This limitation indicates the potential benefit of future studies conducted as full-fledged case studies of the organizational aspects of chatbot implementation, utilizing a broader range of data sources.

Finally, the study being conducted at a single point in time makes it limited in that we may miss insight into the long-term implications of organizational factors. Also, there may be details and changes in the development of the chatbot implementation that are lost in the retrospective accounting of details in interviews with resource personnel in the organizations. Motivated by this limitation, we anticipate future research that track chatbot implementation projects over time to gain further insight into organizational aspects across the entire implementation process and further into stable production.

Conclusion

We have presented findings from an interview study involving resource persons from six organizations that had implemented chatbots for customer service, to gather insight into organizational aspects of such implementation. The research was conducted as a qualitative exploration in response to the limited existing knowledge in this specific area of interest. The findings have provided new insight into organizations' motivations for implementing such chatbots, related success criteria, and relevant organizational factors.

Our findings are in line with existing theory on implementation of digital innovations. At the same time, these provide new knowledge on organizations' motivations and views on success criteria for the implementation of chatbots for customer service. The five organizational factors of particular interest in the study provide new understanding of the work and team organization required to take advantage of chatbots for customer service, including how to set up for the emerging role of the AI-trainer, in addition to improved insight into aspects of change management, competency management, organizational resources, and performance measures. As such, our findings complement the existing literature on digital innovation implementation as well as the current literature on chatbot research. We hope that the findings will motivate further research into this increasingly relevant knowledge area.

Acknowledgement

We express our gratitude to our study participants. The contribution of the second author was supported by the Research Council of Norway, grant no. 282244.