Why do newly industrialized economies deter to adopt responsible research and innovation?: the case of emerging technologies in Korea

ABSTRACT

Technologically advanced countries have recently highlighted the responsible research and innovation (RRI) approach, which aims to increase the positive effects of innovations by intervening in the early stages of technological development. It is necessary to prepare for unexpected impacts that might occur as emerging technologies continue to develop. Korea has a newly industrialized economy; therefore, it is important to examine potential RRI implementation in Korea, as RRI remains a controversial topic. Korea also lacks policy experience regarding key RRI elements, which may present barriers to future RRI implementation. This study attempts to advance RRI in Korea by using the analytical hierarchy process approach to identify and prioritize the expected RRI barriers to autonomous vehicles and biotechnology. The results indicate that economic and policy/politics barriers are the most important. Specifically, the increased innovation costs and lack of economic incentives are the most significant barriers facing RRI.

KEYWORDS:

• Responsible research and innovation
• analytical hierarchy process
• barrier
• emerging technology

Introduction

In recent years, social and ethical contribution of research and development (R&D) has become significant. Technologically advanced countries in Europe have started using the responsible research and innovation (RRI) approach to prepare for the social impacts of emerging technologies (Burget, Bardone, and Pedaste 2017). Few newly industrialized economies (NIEs) such as Brazil, China, and South Korea (hereafter referred to Korea) have rigorously addressed issues related to RRI (Åm 2019; Glerup, Davies, and Horst 2017; van Hove and Wickson 2017). NIEs have concentrated on innovation-based economic growth, ignoring the social and ethical aspects of R&D. However, as these countries achieved successful industrial and economic growth, they started to consider the implications of innovation for various social issues. Korea is one of the most successful NIEs, and has concentrated on R&D activity. In 2017, Korea's R&D intensity (i.e. the country's expenditures on R&D as a percentage of gross domestic product) was the highest in the world at 4.55% (OECD 2019). Furthermore, Korea is an example of a catch-up economy that successfully developed through leapfrogging (Lee and Lim 2001), which has raised concerns regarding the social impacts of innovation in emerging technologies.

Nevertheless, the literature on social effects in terms of the risks associated with emerging technologies is new and thus limited (Suh 2019). As Korea lacks policy experience on key RRI elements, and since RRI remains controversial in terms of both its concepts and implementation, it is expected that there will be barriers to overcome at various levels in regard to the practical application of RRI (Burget, Bardone, and Pedaste 2017). It also has been argued that cross-national contexts play an important role in the application of RRI. Existing studies have shown that perceptions of RRI differ across Europe depending on each specific country's (developed or less-developed) innovative environment (Lukovics et al. 2017). Therefore, before RRI can be effectively applied in Korea, it is important to learn from Europe's experiences, while simultaneously recognizing Korea's unique context and identifying the specific barriers it faces regarding RRI adoption.

Due to the unintentional negative social effects of emerging technologies, RRI views interventions in the early stages of technology development as especially important to prepare for offsetting the benefits of technology development (Mitchell, Brown, and McRoberts 2018). However, the compulsory nature of these interventions, which take the form of regulations, brings cost consumption to various actors (e.g. firms). If these interventions have social benefits, policymakers push for them, but resistance to policies can arise. In addition, since it is hard for policymakers to remove barriers at once, they need a step-by-step strategy to support firms’ engagement in RRI. However, it has been argued that the costs of introducing regulations such as RRI create offsetting efforts and bring about new innovations, which in turn serve as a benefit (Porter and van der Linde 1995). As a result, RRI should be recognized as a competitive opportunity and RRI should be adopted in a step-by-step manner. However, real-world actors are not aware of this because they are not prepared to implement their innovation strategies for RRI (Ceicyte and Petraite 2018).

This study aims to analyze barriers to the introduction of RRI in Korea with respect to policy and social acceptability. We address strategies for adopting RRI by considering stakeholders who engage in and value RRI. This study applies the analytical hierarchy process (AHP) to proactively and comprehensively investigate the barriers to RRI adoption in Korea. AHP is a widely used tool for formulating and analyzing decision-making processes and frameworks, and it is advantageous because it facilitates pairwise comparisons to estimate the weight of certain criteria and to rank various alternatives (Saaty 1990, 2008). Although previous studies have identified the various obstacles that RRI must overcome, these barriers were neither weighted nor ranked to assess their relative importance and impact. Therefore, this study makes a novel contribution by prioritizing 13 barriers within five categories that need to be resolved to implement RRI in Korea.

We conducted a survey by targeting professionals with a general understanding of Korea's R&D systems and technologies. The participants in the survey were technical experts working in policy, academia, and firms. While research on firms has often been limited to more detailed management situations, this study examined fewer specific conditions, assuming that the overall perception would be that Korea is still in the early stages of the introduction of RRI. Since the RRI concept has not universally proliferated in Korea, it is only possible to predict the future impact of technology by discussing specific technologies. Therefore, we selected autonomous vehicle (AV) and biotechnology (BT) as the target technologies and examined the effects of the characteristics of these emerging technologies on rank of barriers to RRI adoption in Korea. AVs have the risk of system errors and disputes about responsibility between operators in the event of an accident (Cui et al. 2019; Keeling 2018; Poel et al. 2017; Sheehan et al. 2019), and BTs may lead to adverse effects and bioethical infringement risks (Gartland and Gartland 2018).

This paper is organized as follows. Section 2 reviews the RRI literature and Section 3 classifies the barriers to RRI adoption. Section 4 describes the AHP method used for the analysis. Section 5 provides results and a discussion of the findings. Section 6 presents conclusions.

Literature review

Definition of responsible research and innovation

The RRI approach has succeeded in effectively presenting the importance of responsible research to society (von Schomberg 2013). As part of Horizon 2020, which is the European Union (EU)'s program for supporting research, the EU is strongly advocating for RRI and related academic research is being actively conducted (Chatfield et al. 2017; Guston et al. 2014). Simultaneously, the conceptual basis of RRI is developing from administrative and academic perspectives (Burget, Bardone, and Pedaste 2017). First, from the administrative perspective, the European Commission (EC)'s definition emphasizes inclusiveness and participatory governance, and it presents RRI as an approach that provides a certain direction for research and innovation, not just a process (Burget, Bardone, and Pedaste 2017). As the second of the administrative definitions, von Schomberg (2013)'s formulation encompasses a comprehensive range of aspects and elements, including inclusiveness, participation, expectations, social satisfaction, and ethical acceptability, and it is closely linked to the processes and values of EU policies; as such, it is a commonly applied definition of RRI in the literature.

Academic definitions mainly focus on the dimensions of RRI (see details in Appendix A). Stilgoe, Owen, and Macnaghten (2013) proposed that RRI has the four dimensions of ‘anticipation,' ‘reflexivity,' ‘inclusion,' and ‘responsiveness.' The acronym ‘AIRR' has been used to refer to these dimensions. Since then, these dimensions have been adjusted in discussions of RRI policy to ‘anticipate,' ‘reflect,' ‘engage,' and ‘act,' or ‘AREA.' This terminology draws attention to another step in the process of negotiating and developing the RRI lexicon, and the academic definition of RRI has an analytically flexible framework (Murphy, Parry, and Walls 2016).

This study follows the RRI definition suggested by von Schomberg (2013), and focuses on societal actors who face difficulties in adopting RRI but care about ethical acceptability, sustainability, and societal desirability of the innovation process. The concept of von Schomberg (2013) is more suitable than academic definitions for illustrating RRI in real-world contexts, and therefore it has been widely used in previous RRI research.

Barriers to responsible research and innovation implementation

Though RRI has become important in recent policy research, previous studies have not fully investigated the application of RRI (Burget, Bardone, and Pedaste 2017; Lubberink et al. 2017). Since the concept is new and stakeholders have not had sufficient time to understand and prepare for its implications, various barriers are expected to hinder the actual applications of RRI (Chatfield et al. 2017), a likelihood identified by numerous researchers. First, as mentioned earlier, specific definitions and implementation methods of RRI are still under development (Burget, Bardone, and Pedaste 2017; Carbajo and Cabeza 2018; Chatfield et al. 2017). Second, the potential benefits that might arise from RRI remain uncertain (Chatfield et al. 2017; Genus and Iskandarova 2018). Third, it is necessary to explore how scientists deal with social concerns related to research practices. To do this, RRI disciplines must first develop a shared language of responsibility with scientists and more actively address the political context of modern scientific research (Åm 2019; Glerup, Davies, and Horst 2017). Fourth, the success of RRI varies depending on the social, ethical, and environmental context (Davis and Laas 2014; Lukovics et al. 2017). While several studies have been conducted on RRI, such research is often limited to conceptual presentations and thus contains insufficient empirical evidence on the presence and effects of these barriers. Therefore, it is necessary that numerous empirical studies on this topic be conducted in the future to address these gaps.

Implementation context of responsible research and innovation

In order to implement RRI, policymakers need to consider two factors: the preferred and feasible level of RRI, and the national context. First, the benefits and costs of implementing RRI vary depending on the RRI target level (Paredes-Frigolett 2016). Although research on the framework of RRI levels based on Stilgoe, Owen, and Macnaghten (2013)'s definition has been conducted, previous studies have rarely investigated the preferred level of RRI, which depends on the capabilities of the adopters, relationships with stakeholders, and similar factors (Gianni and Goujon 2014; Mei and Chen 2019; Paredes-Frigolett 2016; Stilgoe, Owen, and Macnaghten 2013). Second, research has been conducted on the implementation of RRI in contexts other than Europe, but there has been insufficient discussion of the factors that should be considered as differentiators from Europe (Arnaldi et al. 2015; Dalziel et al. 2018; Davis and Laas 2014; Lubberink et al. 2019).

It has been argued that the cross-national context plays an important role in the application of RRI, which seems to be developing successfully because developed European countries are experienced in implementing policies relating to key RRI elements (Lukovics et al. 2017). For example, these countries have developed a variety of technology impact assessment methods, such as real-time technology assessments, and they have gained experience with ethical reflexivity systems by evaluating the effects of certain ethical, legal, and social aspects (Guston and Sarewitz 2002; Zwart, Landeweerd, and van Rooij 2014). Furthermore, since the late 1970s, European countries have made concerted efforts to consider the opinions of the public and non-governmental organizations when making decisions related to science and technology (Landeweerd et al. 2015). This policy experience is expected to have a positive impact on the use of RRI, even if it was previously considered unfamiliar or irrelevant to the RRI discourse (van Åm 2019; Glerup, Davies, and Horst 2017; Hove and Wickson 2017). However, according to the literature, even within the EU, researchers in developed countries expressed interest in the interaction between technology and society, while researchers in non-developed countries expressed the need to concentrate on securing research funds and had a low understanding of the need to consider the social context of technological development (Lukovics et al. 2017). There were marked differences in perceptions. Thus, it is necessary to consider the innovation environment of each specific country to augment the effectiveness of RRI. In addition, it has been argued that the technology level (high technology vs. low technology), culture, and material barriers to innovation affect attitudes toward RRI implementation and should be considered (Giulio et al. 2016; Hartley et al. 2019; Hoop, Pols, and Romijn 2016).

With the spread of RRI around the world, it has been argued that, especially in developing countries such as China and India, RRI must take into account the technological, social, and political context. A study reviewing governments, businesses, and publicly funded scientists as entry points for RRI in China pointed out that, despite the existence of developments and best practices for RRI, ‘there lacks an institutional mechanism for dialogue and for exchanges to take place across different levels' (Gao, Liao, and Zhao 2019). Hence, in this context, it is argued that the transformation of innovation requires institutional innovations, especially to promote new ways of communication (Doezema et al. 2019).

It is important to consider a country's research and innovation environment, as it can have a strong impact on the motivation to implement RRI. Korea has an adverse environment to RRI implementation. For example, Korea's technology impact assessment made the criticism that R&D results are not used for relevant policymaking and that the citizen forum is too formal (Suh 2019). Therefore, it is expected that Korea will face RRI implementation barriers different from those faced by advanced European countries. The question of the suitability of the oocyte collection process that emerged during the study of embryonic stem cell culture using human somatic cells in 2004 was an opportunity to reflect on the ethics of life science research and the importance of research ethics for scientists in Korean society. Subsequently, the enactment of the Bioethics and Safety Act in 2005 provided a set of principles for the nation that could amplify social interest and regulate conflict regarding the development of life sciences and these technologies (especially genome and embryo research). Later, through the Korean National Bioethics Committee (KNBC), Institutional Bioethics Committee (IRB), and Common Institutional Bioethics Committee, a systematic bioethics system was established and the role of self-regulation was assigned to researchers. In recent years, it has been argued that these institutions need to adopt a role of discovering ethical issues and publicizing them to the rest of society (I. Baik and Kim 2017; Lee 2017). In addition, discussions are continuing on ways to expand social acceptance by preparing legal grounds for disputes over possible system errors and responsibility between operators in the event of an AV accident (KISTEP 2018). The introduction of RRI will be helpful insofar as it expands the scope of discussions on the social aspect of technology.

Responsible research and innovation for emerging technologies

It has been argued that different technologies can have distinct side effects even within the same country, which indicates that countries need to tailor their RRI implementation strategies to cope with the unique side effects that they may experience (Poel et al. 2017). The emerging technologies for RRI are those that would contribute to forming dominant paradigms within 15 years, such as nanotechnology and genomics. These technologies can provide more opportunities to address social challenges in terms of sustainability and ethical acceptability (Ko and Kim 2020; Scholten and van der Duin 2015). However, they have been disputed from ethical, legal, and social aspects (ELSA) (Ko and Kim 2020; Scholten and Blok 2015; Stahl, Eden, and Jirotka 2013). Mertens (2018) argues three general characteristics of RRI for emerging technologies. First, due to their radical novelty and unpredictability, emerging technologies require assessment. Second, an early assessment is inevitable to impact the trajectory of innovation. Third, unknown factors are required to be anticipated to prepare for unforeseeable events (Mertens 2018). Based on the three factors, emerging technologies open up more opportunities to consider sustainable and ethical issues in addition to technological advancement.

In order to select the technology to be analyzed, we investigated emerging technologies in the EU and Korea. The reviewed documents are (as shown in appendix B): (1) for the EU, the EC Horizon 2020 Societal Challenges Programme and RRI support programs/projects, and (2) for Korea, the government's R&D investment plan and technology impact assessments. This study investigated the developmental status of AVs and BT as representative emerging technologies in Korea with different characteristics and different ethical, legal, and social aspects. From the European point of view, these are of high importance as emerging technologies relevant for RRI (as shown in Appendix B). From the Korean perspective, these are high priorities in the government's R&D investment plan; for instance, they were selected as BIG3 (Future Car, Bio·Health, System Semiconductor), a set of key investment fields in the future (MOTIE 2019), and have been selected as technologies subject to a governmental technology impact assessment (KISTEP 2014, 2015).

The development of AVs is based on the convergence of various information and communications technologies, such as artificial intelligence/big data, high-performance processing software and hardware platforms, and sensor systems. AVs are expected to lead to major advances in the mobility for various reasons, including a reduction in the number of traffic accidents. Korea is trying to develop AV with the goal of commercialization by 2030 (KISTEP 2018). However, the concern has been raised that AV will reduce human control and lead to undesirable consequences (i.e. situations in which responsibility for an accident cannot be specifically assigned) (Poel et al. 2017). Paradoxically, the increased connectivity between infrastructures combined with autonomous driving can be a significant threat to the enormous socioeconomic benefits promised by AVs (Cui et al. 2019; Keeling 2018; Sheehan et al. 2019).

In addition, BTs produce a variety of high value-added products using biological functions and information. Korea's BT industry has a small market size, but a high yearly export growth rate of 11.5%. Despite Korea's weak technical foundation, its BT firms have increased their long-term R&D investments, resulting in certain positive effects such as recent exports of technology (MFDS 2017; MOTIE 2017). However, the combination of factors related to molecular biology, bioinformatics, and new device development may involve convergent or disruptive technologies, which may simultaneously create various ethical, legal, and social issues of concern to RRI. For example, in the field of genomic research, certain challenges have arisen as a result of newly encountered problems, such as privacy protection, data confidentiality protection, ownership of intellectual property in personal data, and prior consent (Gartland and Gartland 2018; Lecuona et al. 2017).

Methodology

Identification of barriers to responsible research and innovation

Because RRI research is new and has not yet been introduced to Korea, the barriers to RRI adoption have yet to be fully understood. Table 1 displays and briefly describes the barriers that we derived from various literature reviews and expert interviews. In total, 14 people were interviewed in this study. The interviewed experts consisted of six social innovation experts (including sociologists), five public R&D planning experts, and three performance experts from public R&D. In addition, these experts were Ph.D. holders with 10–20 years of work experience in the field. As mentioned previously, there are 13 barriers grouped in five categories.

Table 1. Barrier identified from literature

Category	Barrier	Operational definition
Economic Barrier	Increased innovation costs due to RRI implementation	RRI activities that involve various stakeholders incur additional costs and time. For example, anticipation of and reflexivity on the impact of technology, and interacting with stakeholders, both of which are important parts of RRI, can lead to additional costs. (Auer and Jarmai 2018; Chatfield et al. 2017; Genus and Stirling 2018; Hoop, Pols, and Romijn 2016).
	Lack of economic incentive to participate in RRI	Apart from the social demand for RRI, incentives driving RRI implementation are unclear (Chatfield et al. 2017; Dreyer et al. 2017; Gianni 2014; Gurzawska, Mäkinen, and Brey 2017; Lukovics et al. 2017).
Implementation Barrier	Lack of clarity and empirical RRI cases	A lack of both clear RRI concepts and empirical examples can limit the specificity of RRI implementation in a variety of innovative environments (Burget, Bardone, and Pedaste 2017; Carbajo and Cabeza 2018; Chatfield et al. 2017; Davis and Laas 2014; Dreyer et al. 2017; Genus and Stirling 2018; Gianni 2014; Gurzawska, Mäkinen, and Brey 2017; Hoop, Pols, and Romijn 2016; Monsonís-Payá, García-Melón, and Lozano 2017; Paredes-Frigolett 2016; Stilgoe, Owen, and Macnaghten 2013).
	Lack of RRI training and expertise	Learning and training are important so that impacted professionals can both understand the need and develop a motivation for RRI adoption. Training and expertise are also necessary for determining the appropriate RRI implementation methods. In addition, RRI implementation requires the presence of certain professionals who can address the training and expertise needs (Auer and Jarmai 2018; Gianni 2014; Heras and Ruiz-Mallén 2017).
	Lack of organizational structure and suitable RRI culture	Integrating RRI into organizations and systems can be time consuming and labor intensive because RRI implementation requires that the existing practices and procedures change (Auer and Jarmai 2018; Chatfield et al. 2017; Geenhuizen and Ye 2014).
Policy & political Barrier	Increased bureaucracy with RRI implementation	The involvement of social scientists or ethicists in restraining the practice of existing laboratories may make scientists feel as though the administrative and bureaucratic procedures are increasing in complexity (Forsberg et al. 2018; Gurzawska, Mäkinen, and Brey 2017).
	Lack of clear RRI policies and unclear focus on RRI regulation	RRI can be used in private firms’ R&D strategies and in development of policies that support national R&D funding. However, the novelty of the RRI approach makes it unclear what form future policies and regulations will take (Auer and Jarmai 2018; Genus and Iskandarova 2018; Gianni 2014; Landeweerd et al. 2015; Paredes-Frigolett 2016).
	Lack of both trust and political leadership to lead RRI change	The RRI approach focuses on an improved integration of innovation in society through an increased responsiveness embedded in the system. This requires political leadership to drive RRI, and associated societal confidence in the changes it will bring (Dreyer et al. 2017; Wahlster et al. 2015).
Social Barrier	Lack of social acceptance and academic identity of RRI	A successful RRI implementation has to overcome society's existing routines, a capability which is far from certain (Davis and Laas 2014; Genus and Iskandarova 2018; Genus and Stirling 2018; Landeweerd et al. 2015; Lukovics et al. 2017; Stilgoe, Owen, and Macnaghten 2013).
	Lack of experience with bottom-up governance	RRI emphasizes that the consequences of innovation may need to be discussed from social members’ perspective. However, there is a lack of policy experience with this type of social discussion system (Burget, Bardone, and Pedaste 2017; Genus and Iskandarova 2018; Gianni 2014; Gurzawska, Mäkinen, and Brey 2017; Landeweerd et al. 2015; Stilgoe, Owen, and Macnaghten 2013).
	Conflicts of interest and a lack of stakeholder networks	In the absence of an effective network of experienced stakeholders involved in innovation, it may be difficult to find solutions when there is a conflict between the existing innovation approaches and RRI implementation (Auer and Jarmai 2018; Carbajo and Cabeza 2018; Chatfield et al. 2017; Davis and Laas 2014; Geenhuizen and Ye 2014; Genus and Iskandarova 2018; Hoop, Pols, and Romijn 2016).
Technical Barrier	Difficulty in understanding the emerging technologies	Emerging technologies can be difficult to understand, as they deviate from the stereotypes and routines of existing technologies (Landeweerd et al. 2015). This is important because an understanding of the technology is necessary before its externalities can be identified and evaluated.
	Difficulty in identifying the impacts of emerging technologies on society	By definition, emerging technologies are significantly different from existing ones, so it is difficult to predict how their unique methods and practices will impact society. Therefore, a barrier arises because it is difficult to know how to prepare for such impacts (Auer and Jarmai 2018; Chatfield et al. 2017; Genus and Iskandarova 2018; Genus and Stirling 2018; Hoop, Pols, and Romijn 2016).

Data

Multi-criteria decision-making methods, such as AHP, focus on expert knowledge rather than expert numbers (Hessami et al. 2012; Tummala, Chin, and Ho 1997). Existing AHP studies have incorporated various numbers of respondents, from a sole respondent to many (Lee et al. 2012; Saaty 1990; Shin et al. 2009; Singh and Nachtnebel 2016). Additionally, some studies with a large expert group included about 100 respondents to reflect opinions from various fields (Ghimire and Kim 2018; Lee et al. 2012; Shin et al. 2009; Singh and Nachtnebel 2016). Because RRI is an approach to innovation in emerging technologies, it should reflect not only industry, government, and academic opinions, but also technological diversity. Therefore, we selected participants from government, industry, and academia. In this study, in order to overcome the limitations of the questionnaire for Korean experts with a low understanding of RRI, when sending the survey email, detailed information about RRI was also included to help participants understand the background and purpose of this study and to respond to the questionnaire. Then, after respondents became acquainted with the content of RRI, only experts who voluntarily agreed to participate in the survey proceeded to the survey stage. In order to ensure the objectivity and reliability of the respondents, only those with a Ph.D. or at least 10 years of work experience in the relevant field were eligible (24 with a Ph.D. only, 52 with over 10 years of experience only, and 23 with both), and all participants had expertise in either AV or BT. Finally, participation was limited to those with a high level of understanding of technology, as indicated by working in a technology development research institute or research department within a firm. We used 99 responses in the analysis, as we excluded responses that were either incomplete or had less than a 0.1 consistency ratio (Ghimire and Kim 2018). Table 3 presents information on the numbers of respondents for each area of expertise.

Analytical hierarchy process

AHP is a useful way to solve complex decision-making problems. The problem to be solved can be broken down into several sub-problems based on the criteria associated with each sub-problem, and a hierarchical tree can be derived through this process. The AHP model has the advantage of making problems more intuitive and logical so that they are easier to understand (Saaty 1990, 2008; Vaidya and Kumar 2006). Although previous studies have identified certain RRI barriers, they did not rank them or assign them weights (Chatfield et al. 2017). Therefore, we used the AHP method to rank the barriers to adoption. The highest level of this study's AHP model represents the goal of deriving the barriers to RRI implementation in Korea, and the other levels correspond to the categories and specific barriers. Table 2 displays the four steps that are involved in this process (Ghimire and Kim 2018; Luthra et al. 2016).

Table 2. Research process for ranking RRI barriers by using AHP

Stage	Definition	Description
I	Structure the problem and a model	Building a model by decomposing the obstacles predicted for RRI implementation into 13 barriers through 5 criteria
II	Collect data through pair-wise comparison	Collect data based on a questionnaire on a 9-point scale A pairwise comparison matrix is constructed by pairwise comparing factors in the immediate lower hierarchy that contribute to achieving the goals in the higher hierarchy
III	Calculate normalized priority weights of individual criteria	Estimate the relative weights $\mathit{Y}$ of the barriers using the eigenvalue method: $\mathit{Y}=\mathit{A}w,\mathit{w}=\left(w_1,w_2,\cdots ,w_n\right),$where $\mathit{A}$ is a pairwise comparison matrix ($n\times n$), $n$ is the number of criteria, $\mathit{w}$ is the eigenvector ($n\times 1$) called as the priority vector Calculate principal eigenvalue ${\mathit{\lambda }}_{max}$ as follows: ${\mathit{\lambda }}_{max}=\frac{{\mathit{Y}}_1/{\mathit{w}}_1+{\mathit{Y}}_2/{\mathit{w}}_2+\cdots +{\mathit{Y}}_{\mathit{n}}/{\mathit{w}}_{\mathit{n}}}{\mathit{n}},$ Calculate a consistency index (CI) as follows: $CI=\left({\mathit{\lambda }}_{max}-n\right)/\left(n-1\right)$ Check the consistency of expert response by using consistency ratio (CR) as follows: $CR=\frac{CI}{RI},$where RI is random index given by Saaty (1994) and if CR is less than 10%, $\mathit{A}$ is considered to be consistent
IV	Analyze the priority weights and derive solutions	Establishing relative priorities of decision-making by comparing weights obtained for each category and barrier

Table 3. Number of respondents according to technology field and type of affiliation.

Number of respondents		Autonomous vehicle	Biotechnology	Total
Government	Ministry of Industry etc.	12	11	23
Industry	Major company	8	5	13
	Midsize Business	6	6	12
	smaller business	11	14	25
Academia	University	6	6	12
	Research Institute	7	7	14
Total		50	49	99

Results

Category hierarchy results

Table 4 displays the results associated with the five categories of barriers based on expert responses. The results indicate that economic barriers present the largest obstacle to RRI (27.71%). As an NIE, R&D continues to have a significant and important role in Korea. However, due to its recent industrialization and the country's low rate of technological innovation, Korea's net commercialization profit rate is low due to its high dependence on components imported and overseas license fees. Therefore, Korea's economy will be sensitive to the introduction of RRI (as a new R&D- and innovation-related policy).

Table 4. Category rankings for RRI implementation.

Categories	Priority Weight	Rank
Economic barrier	0.277	1
Implementation barrier	0.177	3
Policy & political barrier	0.249	2
Social barrier	0.142	5
Technical barrier	0.155	4

* Note: CR = 0.001

The policy/political barrier category (24.90%) forms the second largest obstacle to RRI adoption. With the exception of certain large firms’ research divisions and government-funded research institutes, Korea's firms remain vulnerable in regard to their abilities to develop their own technologies through R&D. Korea's political climate has a relatively strong influence on the firms' development and commercialization of emerging technologies due to the financial weakness, which leads firms to be more sensitive to the direction of governmental policies. Therefore, the direction of government policy and the degree of regulation can act as barriers in the implementation of RRI.

The third most substantial obstacle to RRI adoption was found to be implementation (17.71%). In order for RRI to be applied at research sites in Korea, it is necessary to develop RRI policies that clearly indicate what the approach intends to accomplish; if not, there may be confusion about the methods and direction of RRI implementation. Therefore, the creation of common guidelines to follow can be helpful (Chatfield et al. 2017). The technical (15.50%) and social (14.18%) barriers to adoption were ranked as fourth and fifth, respectively. The surveyed experts perceived that preparing for emerging technologies’ social impacts and ensuring that society would accept RRI were less important than addressing economic barriers.

Results within barrier categories

Table 5 shows the results obtained by calculating and ranking the weights of the barriers to identify the degree of impact on RRI adoption in each category. The within-category rankings suggest that certain barriers must be addressed first in order for RRI to be successfully implemented. Of economic barriers, the increase in innovation costs due to RRI implementation (16.46%) were the biggest obstacle to implementing RRI in Korea, followed by the lack of an economic incentive to participate in RRI (11.24%). This shows that the experts perceived the increase in direct costs associated with RRI implementation to be more significant than the indirect incentives for adoption. Therefore, it is important to identify the direct costs involved and to suggest measures to support RRI implementation. The two barriers in the economic category were the highest-ranked of the 13 barriers, meaning that the experts recognized them as the largest barriers facing the introduction of RRI to Korea.

Table 5. Barrier rankings within each category.

Category	Barriers	Priority Weight	Rank
Economic Barrier	Increased innovation costs due to RRI implementation	0.165	1
	Lack of economic incentive to participate in RRI	0.112	2
Implementation Barrier	Lack of clarity and empirical RRI cases	0.071	1
	Lack of RRI training and expertise	0.048	3
	Lack of organizational structure and suitable RRI culture	0.058	2
Policy/political Barrier	Increased bureaucracy with RRI implementation	0.049	3
	Lack of clear RRI policies and unclear focus on RRI regulation	0.101	1
	Lack of both trust and political leadership to lead RRI change	0.099	2
Social Barrier	Lack of social acceptance and academic identity of RRI	0.042	2
	Lack of experience with bottom-up governance	0.039	2
	Conflicts of interest and a lack of stakeholder networks	0.064	1
Technical Barrier	Difficulty in understanding the emerging technologies	0.060	2
	Difficulty in identifying the impacts of emerging technologies on society	0.095	1

* Note: CR ≤ 0.001.

Within the implementation barriers, the largest obstacle was the lack of clarity and empirical RRI cases (7.13%), followed by the lack of an organizational structure and culture that are suitable for RRI implementation (5.75%) and the lack of RRI training and expertise (4.82%) (Table 5). Therefore, it would be helpful to provide specific RRI guidelines to address these barriers and then to develop an appropriate organizational structure and training/education for RRI.

In the overall ranking of the 13 barriers, the three barriers in the implementation category were lack of clarity and empirical RRI cases (ranked sixth), lack of organizational structure and suitable RRI culture (ranked ninth), and lack of RRI training and expertise (ranked 11th). Lack of clarity and empirical RRI cases (7.13%, overall ranking 6) showed a similar importance to conflicts of interest and a lack of stakeholder networks (6.40%, overall ranking 7) within the social category. In addition, lack of organizational structure and suitable RRI culture (5.75%, overall ranking 9) showed almost the same importance as difficulty in understanding emerging technologies (6.02%, overall ranking 8) within the technical category. Lack of RRI training and expertise (4.82%, overall ranking 11) showed a similar importance to increased bureaucracy with RRI implementation (4.91%, overall ranking 10) within the policy/political category. These findings illustrate the differences in importance for individual barriers, which will be helpful for optimizing the efficiency of policies aiming to resolve barriers of similar importance when implementing RRI initiatives.

As displayed in Table 5, the largest barrier within the policy/political category was lack of clear RRI policies/unclear focus on RRI regulation (10.13%) followed by a lack of political leadership/trust to lead RRI change (9.86%) and increased bureaucracy associated with RRI implementation (4.91%). The EC, through its funding programs, incentives bottom-up or top-down implementation in institutions that seek funding from the EC's funding calls, and it is expected that Korea will implement RRI in the same way. Studies seeking to expand the conceptual interpretations of RRI are also underway. Therefore, it is important to provide a clearer scope of RRI policy, including various viewpoints, and to support political leadership to lead the introduction of RRI. In the overall ranking of the 13 barriers, the three barriers within the policy/political category were lack of clear RRI policies/unclear focus on RRI regulation (third place), lack of political leadership/trust to lead RRI change (fourth place), and increased bureaucracy associated with RRI implementation (10th place). Lack of clear RRI policies/unclear focus on RRI regulation (10.13%, overall ranking 3) and lack of political leadership/trust to lead RRI change (9.86%, overall ranking 4) had similar rankings to difficulty in identifying the impacts of emerging technologies. These barriers had almost the same importance as impacts on society (9.48%, overall ranking 5). The increased bureaucracy associated with RRI implementation (4.91%, overall ranking 10) showed a similar importance to lack of RRI training and expertise (5.75%, overall ranking 11) within the implementation category.

Within the social barrier category, the largest barrier was conflicts of interest/lack of stakeholder networks (6.40%), followed by RRI's lack of social acceptance/academic identity (3.90%) and the country's lack of experience with bottom-up governance (3.90%). RRI aims to introduce a social discussion system into the technology development process. Therefore, exchanging information and communicating with various stakeholders are important for successful RRI implementation. Stakeholders include the government, firms, and academic members involved in technology development (a narrow focus), as well as social scientists, humanists, non-governmental organizations, and citizens (a broad focus). Conversations among people who were previously unfamiliar with each other can either have positive effects or lead to dissension when there are conflicts of interest. Korea lacks policy experience in these areas, which may lead to a substantial amount of trial and error with regard to RRI adoption.

In the overall ranking of the 13 barriers, the three barriers in the social category were conflicts of interest/a lack of stakeholder networks (seventh place), lack of social acceptance and academic identity of RRI (12th place), and lack of experience with bottom-up governance (13th place). Conflicts of interest/a lack of stakeholder networks (6.40%, overall ranking 7) showed a similar importance to difficulty in understanding emerging technologies (6.02%, overall ranking 8) within the technical category.

Within the technical barrier category, the most influential barrier to RRI implementation was difficulty in identifying the impact of emerging technologies on society (9.48%) followed by difficulty in understanding emerging technologies (6.02%). Korea's current R&D system focuses only on technological development and its related economic effects, without considering the social and ethical impacts of technology; therefore, it is difficult to predict the future risks, or ripple effects, associated with emerging technologies.

Overall ranking of results by technology type

Tables 6 and 7 present the overall results, as well as results from the individual AHP analyses for AVs and BTs. As shown in Table 6, the results for the five RRI barrier categories were consistent across AV, BT, and overall, with economic and policy/political barriers being the two most important factors. Table 7 shows the results for the 13 RRI barriers, and indicates that the findings for the AV, BT, and overall priorities were nearly identical. The five largest obstacles were the increase of innovation costs due to RRI implementation, lack of an economic incentive to participate in RRI, lack of clear RRI policies/unclear focus on RRI regulation, lack of political leadership/trust to lead RRI change, and difficulty in identifying the impacts of emerging technologies on society. The AV, BT, and overall results showed similar trends, with slight differences in the five RRI categories and 13 barriers.

Table 6. Category rankings for RRI implementation by technology.

Categories	Autonomous vehicle		Biotechnology		Overall
	Priority weight	Rank	Priority weight	Rank	Priority weight	Rank
Economic barrier	0.301	1	0.253	1	0.277	1
Implementation barrier	0.156	3	0.200	3	0.177	3
Policy/political barrier	0.246	2	0.251	2	0.249	2
Social barrier	0.145	5	0.138	5	0.142	5
Technical barrier	0.152	4	0.158	4	0.155	4

* Note: CRs of all cases are less than 0.003.

Table 7. Overall barrier rankings for RRI implementation by technology.

Overall barriers	Autonomous vehicle		Biotechnology		Overall
	Priority weight	Rank	Priority weight	Rank	Priority weight	Rank
Increased innovation costs due to RRI implementation	0.178	1	0.151	1	0.165	1
Lack of economic incentive to participate in RRI	0.123	2	0.102	3	0.112	2
Lack of clear RRI policies and unclear focus on RRI regulation	0.091	4	0.111	2	0.101	3
Lack of both trust and political leadership to lead RRI change	0.103	3	0.093	5	0.099	4
Difficulty in identifying the impacts of emerging technologies on society	0.091	5	0.099	4	0.095	5
Lack of clarity and empirical RRI cases	0.066	7	0.076	6	0.071	6
Conflicts of interest and a lack of stakeholder networks	0.069	6	0.059	8	0.064	7
Difficulty in understanding the emerging technologies	0.061	8	0.059	9	0.060	8
Lack of organizational structure and suitable RRI culture	0.049	10	0.068	7	0.058	9
Increased bureaucracy with RRI implementation	0.051	9	0.046	11	0.049	10
Lack of RRI training and expertise	0.041	11	0.056	10	0.048	11
Lack of social acceptance and academic identity of RRI	0.039	12	0.039	13	0.039	12
Lack of experience with bottom-up governance	0.037	13	0.040	12	0.039	13

* Note: CRs of all cases are less than 0.005.

We calculated the overall weight and rank of the barriers with regard to the extent to which they interfere with RRI implementation by multiplying the weights of each category by the priority weights of the barriers. Table 7 displays the overall ranking of all 13 barriers, and the five largest barriers were increased innovation costs due to RRI implementation (16.46%), lack of an economic incentive to participate in RRI (11.24%), lack of clear RRI policies/unclear focus on RRI regulation (10.13%), lack of political leadership/trust to lead RRI change (9.86%), and difficulty in identifying the impact of emerging technology on society (9.48%). From these results, it can be inferred that Korea must provide proper financial support and clear policy directions in order for RRI to be successfully implemented.

Discussion and conclusion

R&D has played a significant and important role in Korea in the past and continues to do so today. Therefore, Korea needs to prepare for the future social impacts of emerging technologies. However, the debate on RRI implementation in Korea has just started. RRI, which recently emerged in technologically advanced European nations, aims to augment the positive effects of innovation in the early stages of technological development. RRI remains controversial in terms of its concepts and practices, and Korea lacks policy experience with regard to key RRI elements. Therefore, these problems may present a barrier to future successful RRI implementation in Korea.

The purpose of this study was to provide a preliminary perspective on factors that might impede the implementation of RRI in Korea. To this end, we first conducted an extensive literature review and consulted with experts to identify various possible barriers to RRI adoption. As a result, we identified five barrier categories (economic, implementation, policy/politics, society, and technology) with a total of 13 barriers across these categories. Although previous studies have identified the various obstacles that RRI must overcome, these barriers had not previously been weighted nor ranked to indicate their importance and relative impact. Therefore, this study focused on ranking these obstacles and then used the AHP method to weigh and prioritize the obstacles to offer advice for Korean professionals who have a general understanding of the country's R&D systems and technologies. The results indicate that the two most important categories of barriers are economic and policy/politics. More specifically, the two most important obstacles were found to be the increase in innovation costs due to RRI implementation and the lack of an economic incentive to participate in RRI. The government could consider operating a financial support plan by establishing an institutional foundation through the introduction of an RRI certification system and by providing incentives to firms that introduce and implement the RRI certification (Gurzawska, Mäkinen, and Brey 2017). Even though the level of Korea's RRI implementation is considered to be high (Ko and Kim 2020), it still pursues economic growth based on technology improvement due to its path dependency. In other words, when applying RRI in NIEs, the economic and technical barriers must be solved first, and then the remaining three categories of barriers can be addressed.

This study provides four theoretical contributions. First, this study shows the priorities of factors impeding the application of RRI to NIEs in non-European countries, as represented by Korea. The results obtained from our research can be seen as reflecting the characteristics of a country with high R&D intensity, but low social connection of science and technology. Second, our empirical contribution is to rank pre-defined RRI barriers according to survey responses from researchers working in industry, government, or academia in South Korea. This study presents new empirical evidence by developing categorized barriers and setting weights for each barrier (based on the barriers identified in the existing literature). Third, the AV or BT targeted for analysis have very different characteristics, and this analysis is meaningful in that it covers a very wide range of technologies. Finally, in terms of our methodological contribution, we demonstrate AHP method can be used to identify and prioritize the expected RRI implementation barriers, as this approach allows researchers to intuitively and logically solve complex decision-making problems.

This study makes two policy contributions. First, this study facilitates discussion on how to overcome the identified barriers. Ranking the obstacles will also help Korea to overcome obstacles to RRI implementation. In addition, the findings of this study suggest that it is most important to prepare alternatives to the economic and policy/political barriers, which emerged as most impactful in our research. These results indicate that it is necessary to prepare the proper level of financial support and to clarify policy directions in order to overcome the obstacles facing RRI implementation. In addition, it might be necessary to encourage active communication among the various stakeholders to gather diversified perspectives in regard to overcoming the barriers to successful implementation of RRI. Furthermore, sufficient preparedness and efforts to anticipate potential conflicts of interest between stakeholders can be useful in countries where R&D is important.

Second, this study analyzes the characteristics of the emerging technologies from examples of RRI implementation in Korea. No significant differences were found between the AV and BT in regard to the importance of the RRI obstacles. This finding could be important for promoting appropriate RRI policies because the selection and intensity of RRI policy implementation (e.g. the calculated RRI budgets) do not need to be dependent on characteristics of the technologies/industries.

This study has three limitations. First, because the analytical method used was based on expert perceptions, we were unable to estimate the quantitative figures needed to solve the obstacles. For example, it may be necessary to calculate certain values, such as the estimated costs of tackling economic barriers, to implement RRI policies. We therefore recommend that further research be conducted to estimate these figures. Second, while our results can guide future research in other countries/regions, this study focused exclusively on Korea and thus cannot be generalized to other regions and contexts. Third, this study focused on only two emerging technologies, AV and BT, and it is therefore necessary for future research to explore additional emerging technologies such as robots, artificial intelligence, and renewable energy.

Ethics approval

The questionnaire and methodology for this study was approved by the SNU Institutional Review Board of the Seoul National University (IRB No. 1907/002-011).

Consent

All study participants provided informed consent.

Acknowledgements

This paper was modified and developed from the Ph.D. thesis of the first author.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Eunok Ko (Ph.D. in Engineering) is a principal researcher in Korea Evaluation Institute of Industrial Technology (KEIT). Her research interests are Technology and Society Convergence, and Technology Commercialization. She has studied the impact of emerging technology on society for her doctoral thesis.

Jungsub Yoon (Ph.D. in Engineering) is an associate research fellow in Science and Technology Policy Institute. His research field includes Innovation Theory, Industrial Transition, Patent Network Analysis, and Agent-Based Modeling (ABM). His current research focuses on Digital Transformation, Artificial Intelligence Policy, and Theorizing Technological Evolution.

Yeonbae Kim (Ph.D. in Engineering) is a Professor in the Technology Management Economics and Policy Program (TEMEP) and Graduate School of Engineering Practice at Seoul National University (SNU). His interests are on Technology Commercialization including technology protection, alliances, financing, and licensing, Technology Innovation System, Energy and Environmental Economics, and Information Economy. His work has previously appeared in journals, including Science and Engineering Ethics, Technovation, Business Strategy and Environment, and Journal of Technology Transfer.

Appendices

Appendix A: RRI dimensions

Table A.1. Dimensions and relevant tools for RRI implementation.

Dimension	Definition	Representative techniques and approaches	Factors affecting implementation	Research
Anticipation	A priori analysis that predicts and analyzes the expected or unintended impacts of technological developments and innovations	Foresight	Engaging with existing imaginaries	Martin (2010)
		Technology assessment	Participation rather than prediction	Guston and Sarewitz (2002)
Reflexivity	Reviewing the fundamental objectives, motivations, potential impacts, and how benefits are distributed of technology development and innovation to meet regulatory and ethical standards	Embedded social scientists and ethicists in laboratories	Enlarging or redefining role responsibilities	Selin (2011)
		Ethical technology assessment	Reflexive capacity among scientists and within institutions	Swierstra, Stemerding, and Boenink (2009)
		Codes of conduct	Connections made between research practice and governance	Schomberg (2013)
Inclusion	A collective deliberation and discussion process through the inclusion of the public and various stakeholders to restructure possible problems and introduce a broad perspective.	Consensus conferences	Questionable legitimacy of deliberative exercises	Kulve and Rip (2011)
		Citizens’ juries and panels	Need for clarity about, purposes of and motivation for dialogue	Guston and Sarewitz (2002), Guston (2014)
Responsive ness	Inclusion of iterative and comprehensive learning processes and direction settings in an effort to internalize the process of prediction, reflection, and participatory deliberation into the innovation process.	Regulation	Science-policy culture	Roger et al. (2015), Watermeyer (2010)
		Standards	Institutional structure	Voegtlin and Scherer (2017)
		Niche management	Institutional cultures	Schot and Geels (2008)
		Stage-gates	Intellectual property regimes	Macnaghten and Owen (2011)

Appendix B: emerging technologies

Table B.1. Emerging technologies studied in this research

Nation	Program/Project	Research	Emerging technology
EU	EC Horizon 2020 Societal Challenges	Owen (2014)	Health (biotechnology), Integrated transport (autonomous vehicle)
	Socio-Technical Integration Research (STIR)	Smolka (2020)	Neuroscience (biotechnology)
		Flipse, van der Sanden, and Osseweijer (2013)	Biotechnology
		Flipse and Puylaert (2018)	Autonomous vehicle
	Engineering and Physical Science Research Council (EPSRC)	Cock Buning and de Bruin (2017)	Autonomous vehicle
		Szollosy (2017)	Robotics (autonomous vehicle)
		Nulli et al. (2018)	IT (autonomous vehicle)
		Taylor and Woods (2020)	Synthetic biology (biotechnology)
	Dutch Responsible Innovation (MVI)	Hoop, Pols, and Romijn (2016)	Biofuels (biotechnology)
		Pols and Romijn (2017)	Biofuels (biotechnology)
	ELSI	Kurihara (2011)	Biotechnology
		Shineha et al. (2018)	Biotechnology
		Bardill (2014)	Biotechnology
		Meagher and Lee (2016)	Biotechnology
	CTA	Moens et al. (2010)	ICT (autonomous vehicle)
	ETC	Stahl, Eden, and Jirotka (2013)	ICT (autonomous vehicle)
Korea	Government Investment Industry	(MOTIE 2019)	Autonomous vehicle, Biohealth (biotechnology)
	Government technology assessment	(KISTEP 2014, 2015)	Autonomous vehicle, Genetic scissors (biotechnology)

Appendix C: survey

Q1. The current assessment identifies the relative importance of the project's goal, ‘Ranking barriers of RRI implementation in Korea' Please select which criterion (standard) is relatively more important.

Table

Criteria	importance ←								Equal (1)	→importance								Criteria
	Extreme importance (9)	(8)	Demonstrated importance (7)	(6)	Essentialor strong (5)	(4)	Moderate importance (3)	(2)		(2)	Moderate importance (3)	(4)	Essentialor strong (5)	(6)	Demonstrated importance (7)	(8)	Extreme importance (9)
Economic barrier																		Implementation barrier
Economic barrier																		Policy/political barrier
Economic barrier																		Social barrier
Economic barrier																		Technical barrier
Implementation barrier																		Policy/political barrier
Implementation barrier																		Social barrier
Implementation barrier																		Technical barrier
Policy/political barrier																		Social barrier
Policy/political barrier																		Technical barrier
Social barrier																		Technical barrier

Q2. From the ‘economic barrier' point of view, please select which criterion (standard) is relatively more important.

Table

Criteria	(9)	(8)	(7)	(6)	(5)	(4)	(3)	(2)	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	Criteria
Increased innovation costs due to RRI implementation																		Lack of economic incentives to participate in RRI

Q3. From the perspective of ‘implementation barrier,’ which alternative do you think is more appropriate?

Table

Criteria	(9)	(8)	(7)	(6)	(5)	(4)	(3)	(2)	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	Criteria
Lack of clarity and empirical RRI cases																		Lack of RRI training and expertise
Lack of clarity and empirical RRI cases																		Lack of organizational structure and suitable RRI culture
Lack of RRI training and expertise																		Lack of organizational structure and suitable RRI culture

Q4. From the ‘policy/political barrier' point of view, please select which criterion (standard) is relatively more important.

Table

Criteria	(9)	(8)	(7)	(6)	(5)	(4)	(3)	(2)	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	Criteria
Increased bureaucracy with RRI implementation																		Lack of clear RRI policies and unclear focus on RRI regulation
Increased bureaucracy with RRI implementation																		Lack of trust and political leadership to lead RRI change
Lack of clear RRI policies and unclear focus on RRI regulation																		Lack of trust and political leadership to lead RRI change

Q5. From the perspective of ‘social barrier,' which alternative do you think is more appropriate?

Table

Criteria	(9)	(8)	(7)	(6)	(5)	(4)	(3)	(2)	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	Criteria
Lack of social acceptance and academic identity of RRI																		Lack of experience with bottom-up governance
Lack of social acceptance and academic identity of RRI																		Conflicts of interest and a lack of stakeholder networks
Lack of experience with bottom-up governance																		Conflicts of interest and a lack of stakeholder networks

Q6. From the ‘technical barrier' point of view, please select which criterion (standard) is relatively more important.

Table

Criteria	(9)	(8)	(7)	(6)	(5)	(4)	(3)	(2)	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	Criteria
Difficulty in understanding emerging technologies																		Difficulty in identifying the impacts of emerging technologies on society

Appendix D: AHP calculation process

In order to the barriers of RRI using AHP, four stages are used, as shown in Table 2. Each criterion is ranked based on respondents. Then, each category (general criterion) or barrier (sub-criterion) was evaluated at the level of the hierarchy based on respondents’ opinions.

First, categories (general criteria) were evaluated. Each pair of the comparison matrix evaluated by the respondent was integrated by geometric means to form a single paired comparison matrix, as shown in Table D.1.

Table D.1 Respondents opinions for pair-wise comparison for each category (general criterion).

Table

	Economic Barrier	Implementation Barrier	Policy & political Barrier	Social Barrier	Technical Barrier
Economic Barrier	-	1.614	1.127	1.778	1.876
Implementation Barrier	1/1.614	-	0.734	1.284	1.111
Policy & political Barrier	1/1.127	1/0.734	-	1.862	1.585
Social Barrier	1/1.778	1/1.284	1/1.862	-	0.906
Technical Barrier	1/1.876	1/1.111	1/1.585	1/0.906	-

Table D.1 is represented in A and will be used to illustrate how AHP works. (D1) $\mathrm{A}=\left|\begin{array}{ccccc}& 1.614& 1.127& 1.778& 1.876\\ 1/1.614& & 0.734& 1.284& 1.111\\ 1/1.127& 1/0.734& & 1.862& 1.585\\ 1/1.778& 1/1.284& 1/1.862& & 0.906\\ 1/1.876& 1/1.111& 1/1.585& 1/0.906& \end{array}\right|$(D1)

A normalized matrix, N, is obtained by dividing each element of A by the sum of the respective column in A. (D2) $\mathrm{N}=\left|\begin{array}{ccccc}0.278& 0.285& 0.280& 0.253& 0.290\\ 0.172& 0.177& 0.182& 0.183& 0.172\\ 0.246& 0.241& 0.248& 0.265& 0.245\\ 0.156& 0.138& 0.133& 0.142& 0.140\\ 0.147& 0.159& 0.157& 0.157& 0.154\end{array}\right|$(D2)

To find the weight of each criterion $w_i$, W is obtained by calculating the average for each row of N. (D3) $\mathrm{W}=\left|\begin{array}{c}{w}_1=(0.278+0.285+0.280+0.253+0.290)/5\\ w_2=(0.172+0.177+0.182+0.183+0.172)/5\\ w_3=(0.246+0.241+0.248+0.265+0.245)/5\\ w_4=(0.156+0.138+0.133+0.142+0.140)/5\\ w_5=(0.148+0.159+0.157+0.157+0.154)/5\end{array}\right|=\left|\begin{array}{c}0.277\\ 0.177\\ 0.249\\ 0.142\\ 0.155\end{array}\right|,$(D3) where ${\sum }_i^w=1.000$ and $w_i$ stand for each category.

To check for the consistency of the respondents’ opinions, consistency ratio (CR) is used: (D4) $CR=\frac{CI}{RI},$(D4) (D5) $CI=\frac{{\lambda }_{max}-n}{n-1},$(D5) (D6) ${\lambda }_{max}=\sum \mathit{A}W,$(D6) where CI is consistency index, RI is random consistency index, and ${\lambda }_{max}$ is the eigenvalue. We use Saaty (1994)'s RI (Table D.2).

Table D.2 Average RI value according to the order(n) of the matrix

Table

n	1	2	3	4	5	6	7	8	9	10
RI	0	0	0.52	0.89	1.11	1.25	1.35	1.40	1.45	1.49

Based on the Eq. (D.1) and Eq.(D.3), AW is calculated as follows: $AW=\left|\begin{array}{ccccc}& 1.614& 1.127& 1.778& 1.876\\ 1/1.614& & 0.734& 1.284& 1.111\\ 1/1.127& 1/0.734& & 1.862& 1.585\\ 1/1.778& 1/1.284& 1/1.862& & 0.906\\ 1/1.876& 1/1.111& 1/1.585& 1/0.906& \end{array}\right|\times \left|\begin{array}{c}0.277\\ 0.177\\ 0.249\\ 0.142\\ 0.155\end{array}\right|=\left|\begin{array}{c}1.386\\ 0.886\\ 1.246\\ 0.710\\ 0.776\end{array}\right|.\left(\mathrm{D}.7\right)$ CI and CR are calculated by Eq.(D.4-D.6) as follows: ${\lambda }_{max}=\sum \mathit{A}W=\left(1.386+0.886+1.246+0.710+0.776\right)=5.004,\left(\mathrm{D}.8\right)$ $CI=\frac{5.004-5}{5-1}=0.001,\left(\mathrm{D}.9\right)$ $CR=\frac{0.001}{1.110},\left(\mathrm{D}.10\right)$

Since the CR < 0.1, the response A is consistent.

Pair-wise comparisons were also used to rank the sets of barriers (sub-criteria) with respect to their associated category (general criteria) using the same methodology and the results are shown in Table 5. Values for the barriers (sub-criteria) were assigned for each criterion of the category (general criterion), and then multiplied by the corresponding weights and finally summed to give a total score. This process was performed multiple times for each step if the layering model contained multiple layers. In this study, the total score was calculated by multiplying the initial score by the weight for the general criteria, and then multiplying by the weight for the sub-criteria once more (Kabir, Sadiq, and Tesfamariam 2014). The above-mentioned results are based on the qualitative judgements of emerging technologies experts. Their judgments were indicated on the web-based I MAKE IT software (http://imakeit.kr).