“Blockchain in government: toward an evaluation framework”

Abstract

The adoption of a new technology such as Distributed Ledger Technology (DLT) in government is a complex process with numerous potential benefits, but also costs and risks. Early pilots introducing DLT into the public sector show that its potential impact will likely vary depending on the context, including, the type of public service. Even within the same public service, the impact of DLT might be distinct for each of the stakeholders involved (the government, civil servants and citizens, among others). As the public sector is diverse, it is critical to get a proper analysis and understanding of the process of introduction of this technology, which encompasses the different dimensions that play a role in the process. This paper presents an original and multi-dimensional evaluation framework to analyze and compare the benefits, costs and risks of the introduction of DLT in the public sector. It considers a comprehensive set of factors, identified and extracted after conducting a systematic review of the literature, representing potential benefits, costs and risks of DLT in the public sector. These are categorized into four separate dimensions: technological, socio-economic, organizational-cultural, and institutional (legal and political). This evaluation framework has been designed to be used by policy-makers interested in analyzing and comparing the benefits and risks of the introduction of DLT in real-world applications of this technology in the public sector.

Keywords:

• Distributed ledger technology
• blockchain
• government
• innovation policy
• evaluation framework
• public services

1. Introduction

Distributed Ledger Technology (DLT) has recently gained significant attention and investment in various industries, including government. In fact, DLT is considered, alongside others (Pande and Taeihagh, 2023), one of the technologies with the greatest potential for disruption in public administration (Cagigas et al. 2022). DLT, commonly used as a synonym of blockchain, is a decentralized database that enables secure and transparent sharing of information among multiple parties by relying on a shared ledger that is distributed among all peers in the network. This ledger is made up of a series of blocks that contain new data, as well as an identifier (a hash) of all the information introduced in the previous blocks of the chain. Each time data is added to the blockchain (Kassen, 2022), it is written to a new block, which is then sealed and becomes a permanent part of the chain. This process continues indefinitely, with new blocks being added to the chain in a strictly sequential manner. This allows any peer on the network to easily verify the information contained in any block on the chain, making it difficult to manipulate the data stored on the blockchain.

DLT has the potential to streamline processes, reduce costs, and increase trust and accountability in government operations. Potential benefits of DLT in government include increased efficiency, transparency, and security in various processes such as voting, procurement, and citizen services (Cagigas et al., 2021). For example, DLT could enable secure and transparent e-voting systems, reducing the risk of fraudulent activities and increasing voter confidence (Kshetri & Voas, 2018). It could also enable the efficient and transparent tracking of procurement processes, reducing the potential for corruption and increasing accountability. However, the hype surrounding DLT has sometimes led to exaggerated expectations - and the introduction of this technology is not exempt from significant challenges (Park et al., 2021). Potential costs and risks associated with the implementation of DLT in government include the high initial investment and technical expertise required, as well as potential issues related to scalability and interoperability. Moreover, DLT may introduce a significant cost derived from its high energy consumption, which is problematic in terms of environmental concerns (Gabison, 2016). Additionally, DLT can potentially displace existing workers and disrupt established practices and systems. Benefits, costs, and risks of the introduction of DLT in government may vary depending on the sector and the stakeholder considered, and might include a broad range of multiple issues, including technological, socioeconomic, organizational, legal and political factors. It is, therefore, crucial to correctly evaluate and measure the effects of the introduction of DLT in government (Allessie et al, 2019), from a perspective capable of incorporating these multiple factors.

This paper offers a comprehensive evaluation framework of the potential benefits, costs, and risks of the introduction of DLT in specific cases within the public sector, ready to be used by policy-makers. In this way, this paper promotes the generation of systematized information about the advantages and disadvantages of blockchain in the public sector and serves as a guide for public sector leaders contemplating introducing DLT. The applied focus of this paper sets it apart from previous literature reviews on blockchain in the public sector (Ølnes & Jansen, 2018; Batubara et al., 2018; Franciscon et al., 2019; Amend et al., 2021; Cagigas et al., 2021). Moreover, it goes beyond previous attempts to develop an evaluation framework of blockchain applications in the public sector (Lo et al., 2017; Fridgen et al., 2018) in that the evaluation methodology is grounded in a comprehensive systematic literature review where the main potential benefits, costs, and risks of applying blockchain in the public sector have been identified (Cagigas et al., 2021). Hence, this paper contributes to facilitating the work of evaluating these predictions and results and checking whether or not they are fulfilled in real world DLT applications.

To do so, based on the systematic review of the literature, this paper first identifies a comprehensive list of factors representing potential benefits, costs, and risks of the introduction of DLT in government in four separate dimensions: technological, socio-economic, organizational-cultural, and institutional (legal and political). For each of these factors, the evaluation framework developed in this paper identifies an evaluation question and defines a metric able to measure it (Key Performance Indicators, KPI). This set of KPIs will allow for the measurement, analysis, and comparison of the information on the benefits, costs, and risks of the introduction of DLT in government in a multi-dimensional perspective while, at the same time, simplifying the understanding of the innovation process. This evaluation framework can be used by researchers, policy-makers, and practitioners to assess and compare the impact of the introduction of DLT in single-use cases, within specific government contexts.

The rest of this paper is structured as follows. The second section addresses the policy problem around the need of evaluating the introduction of DLT in the public sector. The third section discusses the main factors, representing potential benefits, costs, and risks of the introduction of DLT in government, from which the evaluation framework is built upon. The fourth section presents the evaluation framework. The fifth section concludes.

2. The policy problem: evaluating the effects of DLT in the public sector

The lack of knowledge of the effects of DLT in the public sector, in a multidimensional perspective that comprehends all the potential benefits, costs, and risks, is a significant concern for policymakers who aim to promote the introduction of this technology in the public sector. DLT has the potential to revolutionize many aspects of the government, including the way that governments and public organizations manage and share data, conduct transactions, and engage with citizens (Ølnes et al., 2017; Datta, 2021). However, without a clear and complete approach to the multiple effects that the introduction of DLT may have on a specific context within the public sector, it is difficult for policymakers to make informed decisions about whether and how to adopt this technology. A framework for evaluating the introduction of DLT in the public sector needs to be consistent and comparable across different contexts while, at the same time, it should be flexible: tailored to the needs and information availability in each specific case.

There has been a growing recognition among policy-makers that it is important to develop more robust evidence on the effects of DLT in the public sector. To address this policy problem, public institutions around the world have been investing in research and development to increase our understanding of the potential uses and impacts of DLT. This includes funding research studies and pilot projects to test the effectiveness of this technology in different contexts and sharing the results of these studies with policymakers and other stakeholders. Even though these pilot projects are essential to increase knowledge of the innovation process, they often present a number of challenges that complicate the extraction of definitive insights.

One of the main challenges with DLT is that they are relatively new technologies, and there is still a lack of understanding about their potential uses and impacts. This lack of understanding is compounded by the fact that DLT is complex and technically challenging, which can make it difficult for policymakers to understand their potential effects. Additionally, the use of this technology is often subject to political and regulatory constraints and uncertainties, which can make it difficult to conduct rigorous studies on its effects (Amend et al., 2021).

Another challenge is that the evidence on the effects of DLT in the public sector is often fragmented and inconsistent. This is because there are many different applications of the technology, and the effects of each application can vary depending on the specific context in which it is used. The evidence on the effects of DLT is often based on case studies or pilot projects, which can be limited in scope and may not be representative of the broader public sector (Lindman et al., 2020). While the effects of DLT in the public sector may commonly be case-specific, there is a need for a homogeneous framework in which the evaluation of these effects can be done and compared across different cases. A final challenge is that new technologies often create dependencies on technology and other service providers.

In order to limit these challenges, the use of standardized frameworks and metrics to evaluate the effects of DLT in government should be encouraged. In this regard, some earlier contributions can be mentioned. Lo et al. (2017) proposed an evaluation framework to assess the suitability of applying DLT in different use cases. The framework comprised a list of criteria and a process for practitioners to make informed decisions based on the characteristics of the use cases. Furthermore, the paper evaluated the feasibility of the framework using existing industrial trials. In addition, Fridgen et al. (2018) described the development of an evaluation framework on the potential applications of DLT in the public sector, particularly in the German asylum process. The framework consisted of three levels and eighteen categories of evaluation criteria, spanning technical, functional, and legal domains, and allowed for the specification of use-case-specific key performance indicators.

The contribution of this paper to the emerging, existing literature, is to build an evaluation framework of the introduction of DLT in the public sector which, as a novelty, is based on a comprehensive and updated systematic review of the literature on the potential benefits, costs, and risks of DLT within the specific context of the public sector. This can help to build a more comprehensive evidence base on the effects of DLT technology and can also facilitate the development of best practices for their use in the public sector.

3. The potential benefits, costs, and risks of DLT in the public sector: an assessment framework based on insights from the literature

The evaluation of the introduction of DLT in the public sector requires a framework that captures a number of factors, across multiple dimensions, that play a role in the innovation process. Based on a comprehensive systematic review of the literature, we design an Assessment Framework to encompass and analyze the different factors representing potential benefits, costs, and risks of the use of DLT in the public sector, classified into four dimensions: Technological, Socio-economic, Organizational-cultural, and Institutional. This exercise achieves two objectives. First, the Assessment Framework can be used as a general guide when DLT is considered for implementation in most public services and an assessment is required. Second, the Assessment Framework provides a common, multidisciplinary approach, from which a more detailed Evaluation Framework can be developed. This approach captures the multidimensional perspectives to be considered at the “high” level and establishes the conceptual relations between them.

Figure 1 summarizes our Assessment Framework for evaluating the introduction of DLT in specific cases within the public sector. On the left of the framework is the “critical variable” for assessment, the technology, which is being introduced to the public sector, and the public services affected. In the middle, we find the “conditions” affecting the introduction of the technology, which are organized into three categories: socio-economic; organizational-cultural; and institutional (legal and political). The socio-economic dimension captures the public sector in society and its economy at large, as well as considering its main stakeholders: citizens, firms, and the third sector. The organizational-cultural dimension captures elements internal to the public sector as an organization, including work practices within government and civil servant attitudes. The institutional dimension captures elements associated with the legal, regulatory, and political structures at different levels, including the local, national, and international levels, where applicable.

Figure 1. the Assessment Framework for evaluating the introduction of DLT in the public sector.

Source: elaborated by the authors.

This Assessment Framework brings together the main factors representing potential benefits, costs and risks of the implementation of DLT in the public sector, and organizes them around the four dimensions previously introduced.

3.1. Technological

The most relevant technical factors identified as regards the technological dimension are the following:

• Unified system standards: DLT promises to harmonize technical requirements for the gathering and aggregation of public data. However, the lack of initial technical and regulatory standardization has hampered the communication between different networks and their scalability (Allen et al., 2019). This is a critical issue to solve because instead of a single ledger (such as in the case of the internet), there could emerge multiple public and private platforms that would require some level of interoperability.

• Aggregation of data ledgers: Separate databases or sets of data files called “data silos” can stifle productivity by preventing public officials from getting a “360-degree” view of all data. (Bhatia & Wright de Hernandez, 2019). This can result in service disruptions or poor data-driven decisions. DLT could allow for the concentration of larger portions of information constructing more complete datasets of public data. These large pools of data could be used to guide public policy and enhance the efficacy and efficiency of public services.

• Automation of processes: Tasks conducted by civil servants may also benefit from the reduction of everyday human errors resulting from the automated means of storing data provided by DLT (Allessie, 2019). Once DLT is introduced, the tasks of civil servants in certain public services would change, and focus on developing, maintaining, and governing the DLT application (Ølnes et al., 2017).

• Data integrity: Immutability means that DLTs are based on an append-only data structure. DLT verifies every transaction through a consensus mechanism between nodes ensuring no single party has the unique power to alter it. As soon as a new block of data is verified and introduced in the chain, it is almost impossible to modify (Fan et al., 2019). While this is an attractive feature in many government contexts, there may be others where the difficulty of correcting human error is not a desirable feature.

• Decentralization: DLT is not vulnerable to single availability breaches because data is not stored centrally. Furthermore, each node develops the process in a transparent and accountable manner (Myeong and Jung, 2019). As a result, from a technological standpoint, cybersecurity would be a major benefit for citizens in countries that adopt DLT technology.

• Disintermediation: The trust built on a secure and transparent distributed ledger removes the need to hire, pay, and rely on a third-party entity to oversee transactions (Janssen et al., 2020). Payment networks and money transfer services in the public administration systems are all examples of financial intermediaries that could be drastically reduced. Furthermore, smart contracts can organize simple financial arrangements, ensuring that everyone follows the agreement.

• Traceability: The ability to identify and track the information and events associated with a product or service is referred to as traceability. Due to the immutability of the registry, DLTs allow for complete traceability of transactions from the first-time information was input. Location, application, manufacturing characteristics, and environmental issues are just some of the characteristics and attributes associated with a product that can be traceable. Other benefits of traceability for the government could include authenticity, safety, and accountability across various sectors (Iftekhar and Cui, 2021). Aside from other characteristics, each record of product data could also include information about the labor conditions that were used during production. As a result, traceability could aid in the promotion of better human rights and fair labor practices.

3.2. Socio-economic

The main factors identified as regards the socio-economic dimension are the following:

• Financial efficiency: The introduction of DLT into public services benefits governments by heralding new ways of storing and sharing information that may improve processes. DLT proposes an automated means of storing data in a tamper-evident, secure digital format instead of lengthy, bureaucratic procedures, resulting in a potential reduction of costs (Zhang et al., 2020).

• Public value: A high capital input is a requirement to introduce a DLT system for the first time. Previous research on the availability of DLT in local applications has concluded that the current technological cost of switching to DLT may not outweigh the added security it provides (Gabison, 2016). A correct assessment of the public money invested in each specific case is needed in order to conduct a credible cost-benefit analysis.

• Time efficiency: DLT has the potential to drastically reduce the amount of human effort required to run processes in many public services, resulting in time savings. Additionally, this implies a decrease in common human errors (Allessie, 2019). As a result, DLT has the potential to transform and improve the time efficiency of all public services that involve managing large sets of records and sharing information (both internally and externally) with citizens, businesses, and other sectors.

• Environmental impact: The development of DLT poses a significant cost in terms of its high energy consumption which depends heavily on the specific consensus mechanism in place. Overall, converting recording systems to DLT and scaling them to the scale required to serve large populations could be costly and environmentally damaging (Gabison, 2016; Xie et al., 2019).

• Social and geographical inclusion and participation: The usability of DLT technology remains a major roadblock to widespread adoption. Not only governmental bodies but also several social groups may be unable to immediately benefit from new technological applications due to a lack of knowledge and technical skills (Janssen et al., 2020). Before it is released to the general public, it may be critical to improve user-friendly DLT interfaces and ensure some level of DLT literacy.

3.3. Organizational-cultural

The most relevant organizational-cultural factors identified are the following:

• Government accountability: DLT technology may allow the public to easily monitor the activity. This exposure of relevant information could result in a reduction of non-desirable behavior within the governments regarding administrative procedures. DLT are per se designed for transparency and public monitoring, whereas the public sector often has a hierarchical decision structure, which may hinder to exploitation the full potential of the new technology (Ølnes & Jansen, 2017).

• Reduction of bureaucracy: The use of DLT in government services may reduce the need for paperwork and bureaucratic intervention in administrative processes. For example, Bhatia and Wright de Hernandez (2019) highlight the potential of DLT to reduce the amount of paperwork required to verify credentials in the field of records management. Chang et al. (2019) discuss the potential for DLT to reduce the amount of paperwork and interventions required for international trade. The organizational transformation may also lead to a reduction of common human errors brought on by automated data storage. However, internal resistance to change and the risks associated with it may also arise in an organization, constituting a significant barrier to the introduction of DLT

• Agency coordination: The increasing possibility for coordination is another significant organizational factor identified in the literature. On the one hand, DLT has the potential to improve inter-agency coordination (Cagigas et al., 2021). A government DLT proposal could include a shared ledger of administrative documents that any accredited civil servant could view and extract information from. On the other hand, the use of DLT technology could improve communication and coordination between civil servants and other key players in the delivery of public services. In the field of healthcare, for example, DLT could improve direct communication between physicians and pharmaceutical companies, as well as between physicians and their patients.

• Transparency: Although a single-node, centralized system could be transparent, DLT transparency is based on trust, as no transaction can be manipulated after it is recorded. The rebalancing of power in every transaction where information asymmetry is evident may constitute a benefit of DLT for citizens (Centobelli et al., 2022). Furthermore, in a citizen-to-citizen transaction, it becomes very easy to verify whether one network participant has an exact and unmodified copy of the historical data stream.

• Organizational learning: The lack of necessary skills among civil servants is identified by the literature as a major risk of the introduction of DLT. Because DLT is a complex technology, DLT literacy may be a challenge not only for citizens who use the services but also for civil servants. As a result, government agencies would need to train and hire technical experts in order to develop DLT applications. In addition to professional coders, the public system would need to employ a wide body of lawyers who should be familiar with digital law and disruptive technologies (De Filippi et al. 2022).

• Ownership and technology control: DLT is still a complex technology that requires specialized knowledge for creation and management. A minority of experts dictates the rules of the system and how it is governed: this constitutes an additional risk for citizens and governments. Only a few individuals can modify the code, and there is a risk that the design of the system will represent their interests (Ølnes et al., 2017). Therefore, they could hold dominating powers, diminishing the capacity to integrate enough checks and balances into the DLT network.

• Civil servants’ attitudes: As in the cases of other disruptive technologies, such as artificial intelligence and robotization (Clifton et al., 2020), successful DLT adoption requires workers’ acceptance of the technology. However, the literature shows that acceptance depends on a range of contextual factors (Janssen et al., 2020; Cagigas et al, 2022). Public officials’ opinions about the implementation of DLT can be contextualized as part of the larger literature on workers’ resistance to change, which has found that people’s perceptions of the outcomes of the innovation process have a significant impact on subsequent attitudes toward technology.

3.4. Institutional

The main institutional (legal and political) factors identified are the following:

• Legal compliance: the disruptive properties of DLT data might be legally problematic with respect to current laws. For example, the fact that no one can easily delete certain information due to the immutability of DLT might conflict with several European Union laws such as the 1995 Directive or the GDPR (Han and Park, 2022). This is the case of the right to be forgotten for personal data. Furthermore, it is still unclear what kind of legal recognition will receive the data in the DLT, and whether (and which) additional conditions will be required for it to be recognized as legal. Similarly, how to deal with inconsistencies between DLT contracts and court decisions or legitimacy disputes between DLT and physical parallel systems would need to be determined.

• Privacy: Although encryption and pseudonymization help to protect DLT users’ privacy, the risk of re-identification exists. Despite the fact that each user in DLT is associated with a public pseudonymous address, the transactions could be open to the public, and all network participants would see the information. A growing body of evidence suggests that using transaction details to de-anonymize individuals is possible (Liang and Ji, 2022). However, the more transparent the DLT is, the bigger the risk of re-identification.

• System security compliance: Though security is a major benefit DLT may bring, it also poses a crucial risk, according to the literature: the possibility that "private keys” of the DLT system are stolen, or that other potential malicious and coordinated attacks are made to the network. (Fan et al., 2019). When other consensus mechanisms are adopted instead of “proof-of-work”, as a way to reduce energy and computational needs, the security of the network may get affected since these alternative consensus rules are less strict. Additionally, hackers could take advantage of breaking points caused by poor coding. Finally, if the underlying cryptographic algorithms are broken while the DLT is still in use and cannot be replaced or decommissioned in due time, the security advantage of the technology is lost, and this systemic risk is worth being monitored during the lifetime.

• Trust by design: DLT is not a substitute for institutional trust and institutional infrastructure (Brookbanks and Parry, 2022). In fact, the creation and maintenance of the technological systems in which DLT is based ultimately rely on institutions either through direct management or through externalized services. Countries with higher degrees of good quality public and civil services adopt DLT earlier and more successfully (Reddick et al., 2019).

• Citizen participation: the use of DLT for applications such as e-voting, access to public registries or citizens’ cards could represent an opportunity to enhance citizen involvement and co-production of public services (Mačiuliené and Skaržauskienė, 2021). However, this effect cannot be taken for granted. Citizens might be reluctant to use the technology based on a lack of information as well as lack of specific skills required.

4. An evaluation framework for the introduction of DLT in the public sector

The Evaluation Framework is based on the Assessment Framework described in the previous section.¹ The Assessment Framework defined all the main factors, representing potential benefits, costs and risks of the introduction of DLT in the public sector, identified from a systematic review of the literature in the four dimensions considered (Technological, Socio-Economic, Organizational-cultural and Institutional, including legal and political). For each of these factors, the Evaluation Framework deploys a common evaluation question, which addresses a specific query on whether the potential benefit, cost or risk in question has taken place as a result of the introduction of DLT. Then, each evaluation question is tailored into a KPI, after a vigorous round of multiple discussions amongst users, in our case, amongst those pioneer use cases. Though, for comparability, it is preferable the evaluation question and KPIs are common to different use cases, some tailoring must be accepted, so that the specific circumstances and characteristics of each public sector case is well considered. That means that, for each use case, the research question can be interpreted in the way it fits with each case’s specific context, and the KPI can be adapted in accordance with the characteristics, the needs and the availability of information in each case. The whole set of values obtained for the KPIs would serve to evaluate, in a multi-dimensional perspective, the impact of the introduction of DLT, in a specific use case, and to compare it with other use cases within the public sector context.

4.1. Technological

The set of factors in the technological dimension can be understood as an assessment of the technological pre-conditions for the implementation of DLT. In particular, the KPIs on this technological dimension would focus on assessing whether, for each factor under analysis, the implementation of DLT has been successful with respect to an ideal target implementation or, otherwise, a risk for achieving the expected outcomes of the technology is derived. For each of the technological factors described in the Assessment Framework, the Evaluation Framework develops the following evaluation questions and KPIs:

• Unified system standards. Whilst DLT promises to harmonize technical requirements by providing unified standards, it may occur that open standards and interoperability are blocked by incompatible regulation, legal or other structural barriers (Hu, 2022). On this item, the potential evaluation question addresses: “Has the system moved towards a more unified open system?”. The KPI could measure this by an informed assessment of the number of standards that are being used.

• Aggregation of data ledgers. DLT would allow concentrating larger portions of information by aggregating more complete sets of public data. However, technical barriers to aggregation may remain. In this regard, the evaluation question addresses: “To what extent has aggregation of data occurred?”. The KPI could measure this by the number of data sets integrated from different domains (compared to the baseline level).

• Automation of processes. Whilst tasks conducted by civil servants may benefit from the automated means of storing data provided by the interconnexion between DLT and other technologies, technical barriers may remain and hinder this. For this dimension, the evaluation question addresses: “To what extent have processes become automated?”. The KPI to evaluate this could be the number of management tasks that have become automated (compared to a target level).

• Data integrity. DLT improves data integrity through immutability, although this could not be technically achieved. The evaluation question in this regard addresses: “Has data integrity improved?”. The KPI to measure this could be the ratio of immutable data with respect to the target level.

• Decentralization: The decentralized nature of DLT is critical for ensuring data integrity. Nevertheless, technical barriers may arise. The evaluation question for this dimension inquires: “To what extent has decentralisation occurred?”. This could be measured by the ratio between the number of decentralized files and the total number of files.

• Disintermediation: The trust built on DLT removes the need to hire, pay, and rely on a third-party entity to oversee transactions. However, barriers to disintermediation may remain, in particular since the public sector has rarely seen complete disintermediation. The evaluation question here addresses: “To what extent has disintermediation occurred?”. The KPI could measure the number of trusted third parties avoided (relative to a target).

• Traceability: The ability to identify and track the information and events associated with a product or service provided by DLT may improve public service delivery. However, traceability might not be achieved. The evaluation question in this -regard inquires: “To what extent has traceability occurred?”. The KPI could measure this by the ratio of information that could be tracked relative to the objective ratio.

4.2. Socio-economic

In the case of the socio-economic dimension, the aim of the Evaluation Framework is to evaluate the implementation of DLT by comparing its benefits and its costs, and risks with the alternative situation previously existent. For each of the socioeconomic factors described in the Assessment Framework, the Evaluation Framework deploys the following evaluation questions and KPIs:

• Financial efficiency. The introduction of DLT into public services could provide several advantages, which may result in a reduction of costs. However, it may occur that such a reduction of costs does not take place, if costs outweigh benefits. The evaluation question here inquires: “Have you experienced lower costs?”. This could be measured by an economic quantification of the financial costs and benefits of the implementation of DLT, versus the previously existent system.

• Public value. Public finance may support innovation that is effective and creates value for society, but benefits may not be perceived to compensate for the investment in innovation for the government or society. The evaluation question for this dimension inquires: “Has public finance supported innovation that has created value for society?”. This issue could be measured from a cost-benefit analysis of the implementation of DLT from a social point of view.

• Time efficiency. DLT can reduce the human effort required to run processes in public services, resulting in time savings. However, time efficiency for government and stakeholders may not improve if there is a lack of capacity or responses from these actors. The evaluation question in this regard is: “To what extent has time efficiency improved for the government and stakeholders?”. This could be measured by the number of working hours saved with the new system.

• Environmental impact. The introduction of DLT may provide potential environmental gains. However, it also poses a significant cost in terms of its high energy consumption, and potential environmental costs may exceed the benefits. The evaluation question here inquires: “To what extent do environmental benefits outweigh environmental costs?”. This could be addressed quantifying the energy costs generated by the introduction of DLT against the potential benefits derived from the digitization of the service.

• Social and geographical inclusion and participation: The usability of DLT technology remains a major roadblock to widespread adoption, and failure to reach socially/geographically excluded stakeholders is a risk of the introduction of DLT. The evaluation question for this dimension is: “To what extent have previously excluded stakeholders been included in the public service?”. The KPI on this factor could measure the number of previously excluded agents which are involved in the new system after the implementation of DLT.

4.3. Organizational-cultural

Similarly than for the previous dimension, for the Organizational-cultural dimension, the aim of the Evaluation Framework is, for each factor, to identify the benefits and the costs and risks of the implementation of DLT, in comparison with the previously existent situation. For each of the organizational-cultural factors described in the Assessment Framework, the Evaluation Framework deploys the following evaluation questions and KPIs:

• Government accountability. DLT technology, by allowing the public to monitor the network activity, may introduce positive changes in government culture in the form of higher accountability. However, in precisely those areas where the government is least transparent, the government may avoid DLT. The evaluation question for this dimension is: “To what extent has government culture changed positively?”. The KPI for measuring this issue could be civil servant perception of improvement in government culture from the introduction of DLT, from a survey made to civil servants involved in the process.

• Reduction of bureaucracy: The use of DLT in government services aims to reduce the need for paperwork and bureaucratic intervention in administrative processes, although there is a risk that red tape is not reduced after DLT is introduced. The evaluation question in this regard is: “To what extent has red tape been reduced?”. This could be measured by the number of bureaucratic formalities avoided (with respect to the previously existent system) once the technology is introduced.

• Agency coordination: The improved coordination (inter-agency coordination and communication and coordination between civil servants and other key players) is another significant potential benefit of DLT. However, it may occur that friction between agencies is not reduced as a result of the introduction of DLT. The evaluation question here is: “To what extent has agency coordination improved?”. The KPI to measure this could be the time spent on coordination activities and processes (compared to the previous situation).

• Transparency. DLT may provide a rebalancing of power in every transaction in the benefit for citizens, as a result of transparency. However, sometimes this aspect could not be relevant for citizens or could be even detrimental to public organizations. The evaluation question here is: "Is the transparency that the new system allows positive for the service?”. This could be measured from civil servants’ perceptions on this question, based on a survey asking it to civil servants.

• Organizational learning. As DLT is a relatively new technology and most civil servants lack knowledge on this technology and its potential for public services, government agencies would need to train and hire technical experts in order to develop DLT applications. However, governments may fail in offering suitable training to the workforce or suffer lack of take-up of training. The evaluation question for this dimension is: “To what extent has suitable training been provided?”. This could be measured by the number of learning activities organized for civil servants.

• Ownership and technology control. DLT is still a complex technology that requires specialized knowledge for creation and management, hence there is a risk a minority of experts could concentrate power, dictating the rules of the system and how it is governed. On the contrary, the public sector could ensure “ownership” of the technology. The evaluation question here is: “To what extent can it be said the public sector has ownership of the technology?”. The KPI on this issue would address the ratio of modules where the public sector retains decisive control, with respect to the total.

• Civil servants’ attitudes: As in the cases of other disruptive technologies, successful DLT adoption requires workers’ acceptance of the technology. It may happen that civil servants embrace the technology, or that they reject it. Different configurations of DLT may play a key role in this regard (Cagigas et al. 2022). The evaluation question here is: “To what extent do civil servants embrace the introduction of DLT?”. This would be measured by a question, in a survey to civil servants, asking them: "Would you be willing to introduce the new system into the public service?”.

4.4. Institutional (legal and political)

The set of factors in the institutional (legal and political) dimension can be understood as a post-evaluation of the implementation of DLT in the public sector: that is, an evaluation of whether the implementation of this technology accomplishes the legal and political conditions that are essential for achieving the results obtained from the previous dimensions. For each of the institutional (legal and political) factors described in the Assessment Framework, the Evaluation Framework deploys the following evaluation questions and KPIs:

• Legal compliance. DLT might be legally problematic with respect to current laws and court decisions. The evaluation question in this regard is: “Is the processing built on law compliant standards in your jurisdiction?”. This would be measured by an internal evaluation of the conformity with the existent relevant legislation after it is identified by the team making the evaluation, which could result in an affirmative or in a negative answer to this question.

• Privacy compliance. Although encryption helps to protect DLT users’ privacy, there is still a risk of re-identification and cyberattacks. This could generate a lack of trust, resulting in the lack of use of services based on this technology. The evaluation question on this regard is: “To what extent are privacy requirements adequately met?”. The KPI to measure this is based on ISO 27701 (or GDPR in the case of the European Union) and would evaluate the “average conformity rate” to all requirements and recommendations as assessed by an auditor or in a self-assessment exercise.

• System security compliance. Although security is a major benefit of DLT, it also poses a risk that should be taken into account by the system. The evaluation question here is: “Is the system ready to notify a large-scale cyber security incident?”. The KPI to measure this is based on ISO/IEC 27002 and would evaluate the “average conformity rate” to all requirements and recommendations as assessed by an auditor or in a self-assessment exercise.

• Trust by design. The creation and maintenance of the technological systems ultimately rely on institutions either through direct management or through externalized services. For this reason, DLT requires institutional trust and institutional infrastructure. The evaluation question for this item is: “To what extent do the introduction of DLT affect citizens’ trust in government?” This would be measure from civil servants’ perceptions on whether the introduction of DLT increased, or decrease, citizens’ trust in government with respect to the previously existent situation, from a survey to these stakeholders.

• Citizen participation: the use of DLT represents an opportunity for new mechanisms for citizen participation in government. However, there are several risks in this regard: DLT in public services that have actual users remains rare, DLT may not be fully public, and citizens may resist using mechanisms for participation. The evaluation question in this regard is: “To what extent are citizens participating more in government?” To measure this, the KPI would be the number of end-users informed about and participating in the new system compared to the previously existing before DLT was introduced.

5. Conclusion

DLT has been heralded as the “next big thing” for nearly a decade now. Even though attention around the technology has been significant over the past years, real-world evidence of realized benefits of DLT in government is still hard to find. Existent evidence is fragmented, based on specific case studies or pilot projects, and there is a lack of a homogeneous framework that allows for the evaluation, and comparison of, the benefits, costs and risks of the introduction of DLT in different cases within government activities. This paper provides a multidimensional framework for evaluating the introduction of DLT in government based on a comprehensive review of the literature, which encompasses four dimensions: technological, socioeconomic, organizational-cultural, and institutional (legal and political). This evaluation framework can be of use for policy-makers and practitioners, for researchers, and also for companies providing DLT-based solutions, aimed at obtaining further knowledge and comparing the effects of DLT implementation in various government contexts. Indeed, it has already been applied in the TOKEN project across four, different public sector contexts and the outcome has been published as a Toolkit for the Adoption of DLT in Public Services. As more DLT implementation is evaluated, it will become possible to obtain sufficient empirical data on the consequences of this technology beyond the hype, and to plan its future in the public sector.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research has been carried out with the support of the European Union’s Horizon 2020 Research and Innovation Program through the project TOKEN under Grant 870603. The research has also received the support of the Erasmus + Programme of the European Union (Jean Monnet Action 620296-EPP-1-2020-1-ES-EPPJMO-MODULE); and “Concepción Arenal” research grant program of the University of Cantabria co-financed by the Government of Cantabria.

Notes

1 In the TOKEN H2020 project, the Evaluation Framework was applied across four pilot use cases, representing pioneering cases in the introduction of DLT in government across four EU municipalities. The results have been published as the Impact Assessment (D.4.3.) and the publicly-available Toolkit for Adoption of DLT in Public Services, of the TOKEN H2020 project available at https://token-project.eu/resources/