A Systematic Literature Study to Unravel Transparency Enabled by Open Government Data: The Window Theory

Abstract

The opening of data has been credited for improving transparency and for providing a window on government functioning. Although this relationship is intuitively apparent, it is in fact complex and the mere opening of data might not actually yield transparency. In this paper, a comprehensive model of determinants that enable or impede transparency enabled by open government data and the expected effects have been derived by surveying public administration and information systems literature. Public administration literature tends to be focused on factors such as participation and trust, whereas information systems literature focuses on factors such as user interface, user experience, and data quality. Digital government literature attempts to bridge these elements. The Window Theory is introduced, in order to unify existing models by integrating a broad range of factors within a single model. The Window Theory can be used to develop context-dependent models that are both comprehensive and parsimonious.

Keywords:

• accountability
• anti-corruption
• digital government
• disclosure
• open data
• open government
• open government data
• participation
• transparency

Introduction

Creating transparency is a main driver for the opening of government data (Bertot, Jaeger, & Grimes, 2010; McDermott, 2010p. 45) and is expected to result into benefits like better government decisions (Puron-Cid, Gil-Garcia, & Luna-Reyes, 2012) and accountability (Helbig, Styrin, Canestraro, & Pardo, 2010). Expectations of the benefits of transparency by of open government data are high (see for example European Commission, 2010) and may even be exorbitantly high (Bannister & Connolly, 2011).

Transparency is about creating an insight for someone who is not involved. There is an information asymmetry between the government and the public, as those who are involved in government processes will generally possess more information about their activities than the public (Ahn et al., 2018; Yoon, Zo, & Ciganek, 2011). Transparency is aimed at overcoming this discrepancy in information, thus enabling the public to view what is happening within the government. This is often viewed in the context of the principle-agent theory (Eisenhardt, 1989). In the digital government, transparency portals have created a window to view government functioning, aimed at overcoming the information asymmetry between the government and the public.

Although transparency might be intuitively an appealing concept, people are likely to give different meanings to the concept, with ambiguity as a result. In some cases, the mere opening of data are viewed as a form of transparency, whereas others suggest that there is only transparency if the public is able to understand and interpret the data (Zyl, 2014). Furthermore, transparency is subjective. What is transparent to one person, might not be transparent to another. For example, one person might possess statistical knowledge and be able to interpret the data, whereas another person might lack the required knowledge. Transparency is sometimes even perceived as a “magical concept for everything” (Ward, 2014, p. 45). Despite the ambiguity surrounding this concept, there is an agreement that transparency is a crucial factor for creating an open government and that government data should be opened for this purpose (Zuiderwijk, Gascó, Parycek, & Janssen, 2014). However, simply disclosing more information can actually result in less understanding, more confusion and less trust (Bannister & Connolly, 2011) and might not necessarily result in transparency. Therefore, it is important to understand the elements that influence transparency.

OGD initiatives are often focused on creating transparency (Bertot et al., 2010; McDermott, 2010), however, there is no overview of enabling or impeding factors. Understanding these factors is of paramount importance to advice efforts to create transparency enabled by OGD. Furthermore, there are many factors influencing OGD-enabled transparency. They range from the technical aspects of data such as metadata (Strathern, 2000) and system characteristics such as usability (Bertot, Gorham, Jaeger, Sarin, & Choi, 2014) to organizational factors such as internal resistance to transparency (Navarro-Galera, Alcaraz-Quiles, & Ortiz-Rodríguez, 2016). Although there are many models for transparency in general, there exists no model for OGD-enabled transparency. Researchers focus on different aspects of transparency and there is no overall overview of determinants influencing transparency.

Also, there is a variety of effects of transparency that ranged from accountability (Gandía, Marrahí, & Huguet, 2016) to trust and credibility (Michener, 2015). This diversity of factors suggest that transparency is multi-faceted. The effects of transparency enabled by OGD are not known. The multi-faceted nature of transparency poses difficulties for measurement, theory-building, and testing (Hollyer, Rosendorff, & Vreeland, 2014).

Our research questions are 1) what are determinants influencing OGD-enabled transparency and 2) what are possible effects of OGD-enabled transparency? By answering these questions, this paper aims to develop a comprehensive model of the determinants that enable or impede transparency and to understand the possible effects of transparency in the field of Open Government Data (OGD). The resulting model is labeled the Window Theory. These determinants can help policy-makers and designers of OGD efforts to truly create transparency. Researchers can use this model to select factors that are relevant to them. This paper is structured as follows. In the next section, the structured literature review (SLR) approach is presented, followed by an overview of determinants and effects of transparency. In Section “The window theory”, the determinants and effects found are used to create the model of the Window Theory. The findings and further research are discussed in Section “Discussion and further research". Finally, our conclusions are drawn.

Research method

Our goal is to advance our understanding of transparency in the field of OGD. We opted for conducting a literature review, as there is a large body of work available in the field of transparency. SLR is a “method of making sense of large bodies of information” (Petticrew & Roberts, 2006, p. 2).

Literature review method

The literature review method was based on the Petticrew and Roberts (2006) approach to the literature review, which was extended using the Hsieh and Shannon (2005) approach to content analysis. The latter was used to identify the determinants of transparency and the expected effects. We adopted an instrumental view in which transparency is positioned between determinants and effects. The initial model, that positions transparency in the center, is shown schematically in Figure 1. Determinants are variables that enable or impede transparency, whereas expected effects refer to variables showing the intended and unintended consequences of transparency.

Figure 1. Basic model of transparency.

The literature surveyed included the top 25 journals with an average impact factor higher than 1,0 in the fields of Public Administration (PA) and Information Systems (IS), based on the 2016 Scientific Journal Rank (SJR - Scimago/Scopus). The keywords used for searching were kept broad by including “transparency” and “government” to avoid missing any relevant papers. The terms were only searched in the fields: title, abstract, and keywords. The SLR was conducted in April 2017. Papers published in 2017 were excluded, to avoid any confusion about which subset of papers from 2017 should be included. Searching on “transparency” and “government” returned a large number of papers, as transparency and government are also commonly used words outside the OGD field. For example, chemistry and environmental science use these words within a different context, such as the transparency of glass or water. Some journals did not have a search function and in these cases, the search was performed manually by accessing each paper and searching for the keywords using the search feature for an article. The results show that there is a lot of work available in the financial domain. Budget transparency is based on the use of well-defined measures and financial reports and is guided by regulations (e.g. Mauro, Cinquini & Grossi, 2018). Articles that focused on budget transparency were excluded, as these do not represent the typical challenges faced by OGD (Conradie & Choenni, 2014; Harrison et al., 2012; Janssen, Charalabidis, & Zuiderwijk, 2012). This resulted in an initial selection of 173 papers. The number of papers was further reduced to 73 papers by only including papers published between 2007 and 2017. This enabled us to focus on the papers relevant to the OGD domain.

Content analysis method

After identifying the papers content analysis was used for data analysis. Content analysis can be defined as a “research method for the subjective interpretation of the content of text data through the systematic classification process of coding and identifying themes or patterns” (Hsieh & Shannon, 2005, p. 1278). Content analysis classifies large amounts of text into a number of categories. In each paper, determinants and effects were identified by reading the entire paper and marking each determinant and each effect. The analysis was confronted with heterogeneity in research approaches and context and limited research employing quantitative research methods. Not only different constructs and measurements were used, but also the quality of empirical publications were found to be different and the selection of measurements was found to be context-dependent. This confirmed our starting point that transparency is context-dependent and there is a need to identify determinants influencing OGD-enabled transparency and its possible effect.

As most of the papers are not addressing the OGD field, the determinants and effects needed to be decontextualized from the context in which the study was performed to the OGD field. The content analysis was conducted by both authors and the results were discussed. Initially, all determinants and effects were marked, in order to keep the list broad. This resulted in a long list of factors in which synonyms were found to be used in different papers to depict to the same determinant or effect. It was often impossible to determine whether the authors of one paper used similar definitions of a determinant or effect to other authors. Many papers lack clear definitions and conceptualization and this complicated our SLR. In such situations, it was assumed that they would use the same definitions. On several occasions, the same wording was used in reference to slightly different situations. For example, primary data can refer to raw data that has not yet been processed, but also to data collected from the source and made ready for use. Instead of trying to define each element, we would label them both as primary data. In further research, we suggest that the determinants and effects should be defined in detail.

In many situations where the literature suggested a causal or correlated relationship, the explanatory mechanisms were poorly described or could be challenged. For example, often it is stated that access should be given to raw data. Although publishing raw data enables finding new insights and avoids predefining view, it might not per se result in transparency, as raw data might not easy to understand and use for most people. As such, the mechanisms for creating transparency might not work in all circumstances. In further research, we suggest that empirical research should be conducted to explain the relationship between determinants, transparency, and effects, and to conduct experiments to investigate the causality of relationships.

Model construction

Hsieh and Shannon suggest to keep the number of categories limited to ensure that they are more easy to understand. This can be done by merging similar codes. The researchers merged the codes independently and then discussed them while comparing them to existing models in the literature. After studying the models, the content analysis resulted in 4 clusters of determinants and 1 cluster of expected effects of transparency. Hsieh and Shannon (2005) recommend to create relationships between categories and subcategories based on their concurrence, influence, and consequences. Determinants and effects were cataloged accordingly, including whether a determinant had a positive or negative relationship with transparency. Factors having similar semantics were clustered. After reading the papers, a total of 42 determinants and 8 expected effects were found. Hsieh and Shannon suggest to create a diagram showing the relationships between the clusters and factors. A tree diagram was created using the software Visual Understanding Environment (http://vue.tufts.edu/). This diagram is presented in Figure 2 and was named “The Window Theory”, reflecting the determinants of transparency and their possible effects. The Window Theory is not a tested theory, but merely a comprehensive overview of relationships suggested by the papers that were surveyed.

Figure 2. The window theory: determinants and expected effects of transparency enabled by OGD.

Our SLR depends on previously published research and the availability of these studies. The selection/exclusion criteria limit the number of surveyed papers in order to make the number of papers manageable. Some papers that might have been relevant were missed because they were not published in the outlets surveyed. We excluded literature concerning budgetary data as this is well-structured and well-defined, whereas this is not the situation for most OGD. The authors had to interpret the work of others and to derive what they found to be relevant. Although there is a lot of work available about transparency, there is no uniformity in the measurement and use of constructs. Furthermore, many studies did not provide definitions, or simply mentioned that determinants of effects were relevant. The studies surveyed are diverse and contain different interpretations of determinants and effects. The authors had to interpret the studies and make decisions on how to aggregate them. Sometimes there were discussions about whether determinants should be merged into one or be kept separate. We made this decision based on the relevance of OGD, which might not apply in other situations. In addition, the relevance and significance of factors could not be aggregated. The studies employ different constructs and are conducted in different contexts, which made it impossible to deduce insight into the magnitude of factors. The Window Theory aggregates determinants and effects into a single mode but does not take the contextual factors into account. Some factors might only be relevant to a certain situation.

Findings

A content analysis was conducted on 73 papers to identify the determinants and effects of transparency. Of these 73 papers, 41 (56% of all 73 papers) used a quantitative research method. Only 16 papers include an explicit model associating determinants with transparency and its effects. None of these models contain more than 16 determinants and 3 effects. This confirms our original premise of the need for a unified model. The analyses of papers revealed 4 stages of OGD for transparency development which will be presented next. This will be followed by a presentation of the determinants and effects of transparency.

The evolution of OGD-driven transparency

Figure 3 shows the number of identified papers per year. The survey of the literature shows a steady increase in the number of publications. Most manuscripts were published in public administration (PA) literature (49 papers) accounting for 67% of the total papers, whereas 33% of papers were published in the field of information systems (IS) (24 papers). Digital government papers can be found in both the PA and IS domains, although journals such as Government Information Quarterly are classified in the IS domain.

Figure 3. Frequency of publications per year and area.

From the SLR and content analysis, four phases of transparency research in OGD were identified by looking at the discontinuity. Discontinuity signifies some form of observable change driven by an event, whereas continuity implies maintaining the status quo. Discontinuity helps to find the boundaries for groups of things that are alike (Cook, 1996). The events resulting in discontinuity are derived from the content analysis of the papers and its influence becomes only visible after a while. The first discontinuity is the initiation of Obama’s memorandum of understanding in 2009, resulting in the rise of literature from mainly Western countries. There are many papers referring to this memorandum of understanding published in the subsequent years. The second discontinuity is driven by the creation of Open Government Partnership in 2011. The final discontinuity is less visible as an event, but clearer from the number of papers published. There is an increase in the number of papers in this field which are focused on viewing possible benefits from OGD from a realistic perspective. This phase is less explicit in the literature. The phase is found by papers showing the impediments and disadvantages, whereas the simply stating of advantages still continues.

• 1.  Ex-ante phase (before 2009)

• 2.  Initiation phase (2009-2011)

• 3.  Hype phase (2011- 2013)

• 4.  Realism phase (2014-now)

In the first phase, ex-ante open government data, transparency was given regular attention by researchers. OGD and transparency had no momentum and had gained only limited political attention. Many countries had Freedom of Information Acts (FOIAs) in place, through which citizens could request information (Petticrew & Roberts, 2006). FOIAs allow the public to ask for (partial) disclosure of unreleased information.

The second phase, OGD initiation, is hallmarked by Obama’s Memorandum on ‘Transparency and Open Government’ published in 2009. Before the Memorandum there was limited attention for OGD, even though transparency has always been an important topic for researchers in the field of government and public administration. The Memorandum encouraged active disclosure of public data. This Memorandum resulted in the development of the OGD portals (www.opendata.gov) which offered open data to the public.

This is followed by the OGD hype phase characterized by an unbridled enthusiasm that resulted in many followers. Many local and central governments initiated open data portals to publish their data. In 2011, several countries around the world joined the Open Government Partnership (OGP) to create Open Data Plans and to increase the proactive disclosure of public data (Petticrew & Roberts, 2006). In 2013, the G8 Charter also declared its objective of opening up high-value data from the government (Hsieh & Shannon, 2005, p. 1279). This reflected a change from waiting for a specific request for papers or electronic forms (FOIA) to the pro-active opening of data resulting in massive amounts of data sets in a friendly format (open data, linked data) with free and open access, with the intention of contributing to transparency.

In the fourth phase, named realism, a more feasible ambition has been set and research is initiated to advance OGD efforts in creating transparency. After the initial excitement, scholars identified that data disclosure per se would not lead to proper levels of transparency, accountability, anti-corruption, and the other expected effects. The fourth phase was driven by improving practices and the need for research to deal with limited use. The massive disclosure of data raised the question of whether the opened data had fulfilled the expected promises, for example, and whether transparency enhanced trust, accountability transparency and engagement (Obama, 2009). The number of papers increased substantially, including papers with a technical perspective from IS. At the same time, the term “big data” started to emerge, aimed at using data for the purpose of value creation (Piotrowski & Van Ryzin, 2007).

This research broadened the research in transparency across the board from technical to societal issues. The more technical papers demanded data in standardized, electronic formats using web services following Tim Berners-Lee’s 5 Star model (Obama, 2009). At the same time, the indiscriminate opening of data resulted in privacy concerns Coglianese (2009) and concerns about the proper use of data (Harrison, Pardo, & Cook, 2012). This more technical focus resulted in new concepts such as “transparency-by-design”, in which the opening of data is integrated within the design of information systems (Janssen et al., 2017).

Determinants of transparency

The content analysis revealed many and diverse determinants of transparency for OGD. For each of the determinants, it was described whether the factor enabled or impeded transparency and the explanatory mechanism as shown in Table 1. The first column of Table 1 shows the total number of studies in which the determinants were found. The second column gives this number for only the quantitative studies. This is followed by the determinant name, the description of the determinant, the description of the effect(s). The final two columns contain the references to the studies reviewed.

Table 1. Types of possible effects of transparency.

Expected Effect	Description	Type of effect
Accountability	Transparency is expected to encourage governments to account better for their actions. In practice, accomplishing this can be difficult. Sometimes transparency is viewed as a sine qua non for accountability, although there is no agreement on this.	Positive
Trust and credibility	Transparency can result in more trust, trustworthiness, credibility and legitimacy. Occasionally, a decrease in trust is mentioned, resulting from this openness and seeing how the government works or does not work.	Positive and negative
Civic engagement and participation	Transparency should result in civic engagement as OGD should enable citizens to engage and participate, by providing information. In the literature there is discussion about the limited level of engagement.	Positive
Efficiency	Transparency can help to create efficiency and reduce costs. For example, the improvement of public procurement, allocation of budget and healthcare. Conversely, investments are needed to help create transparency.	Positive
Governance and political turnout	Transparency should result in better governance and political turnout (votes). This is heavily debated in recent literature (fake news).	Positive
Anti-corruption and fraud	Transparency should result into less corruption by reducing bias and the misuse of political power and public funds.	Positive
Overcoming information asymmetry	Transparency should result in overcoming information asymmetry, but some data might not be accurate (low information quality) or manipulated.	Positive and negative
Privacy	Transparency can affect privacy and even violate data protection legislation. For example, the European data protection act requires that civil servants’ identities should not be revealed. This imposes limits on transparency.	Negative

Figure 4 shows the determinants and the number of papers mentioning the determinants. From this figure, it becomes clear that some factors are only mentioned a few times, such as urban areas and types of organization, while open and free access were mentioned most frequently. Some factors are viewed as conditions for creating transparency, whereas others are enablers or inhibitors of transparency. For example, a condition for transparency is completeness, as missing information results in less transparency. Metadata is a typical enabler for the creation of transparency, whereas lack of usability inhibits transparency. Making a distinction between quantitative and qualitative studies enabled us to compare whether different determinants were found in both kinds of studies. Only “technology Literacy” was found in 2 qualitative studies, but not in any of the quantitative study, whereas all the other determinants were found at least one time. Although there are some variations in the number occurrences, the differences between quantitative and qualitative studies are limited.

Figure 4. Frequency of determinants per area.

The variety and diversity of determinants also show the complexity of creating. It is both challenging and costly to satisfy all factors, and may even be unnecessary (Roberts, 2002).

The question remains: which of these factors are conditions that have to be satisfied, which are most influential and which are nice to have. The SLR does not shed a lot of light on this, as there is no consensus in the literature. Different authors suggest that different factors are of importance.

Determinants were found to have different influences on the desired effect. For example, for anti-corruption and fraud, 35 determinants out of 42 were found to have an effect. In contrast, only 27 determinants were found to be related to accountability (see Table 1). The SLR shows that creating transparency in order to reduce fraud is influenced by other determinants than creating transparency for accountability. This suggests that the form of transparency needed is dependent on the intended effects of transparency.

Expected effects of transparency

Transparency can offer many benefits, but also some unintended or even adverse effects, as listed in Figure 5. In the literature, positive effects are emphasized, while negative effects have been given less attention. Examples of negative effects include violation of privacy due to the release of data containing privacy-sensitive information (Harrison et al., 2012), or the fact that the release of large amounts of OGD can result in information overload and errors (Grimmelikhuijsen, 2012). The achievement of either negative or positive effects is highly dependent on how information is released and shared as Table 2 – Types of Possible Effects of Transparency shows.

Figure 5. Overview of expected effects and areas.

Table 2. List of determinants of OGD-enabled transparency.

Frequency (total)	Frequency for quantitative studies only	Determinant name	Determinant description	Description of effect	Sources all studies (Papers ID can be found in Appendix A)	Sources for quantitative studies (Papers ID can be found in Appendix A)
44	24	Free Access	Data sets can be directly accessed at no cost.	Data availability is a condition for transparency.	1, 3, 4, 5, 9, 10, 11, 20, 21, 23, 24, 25, 27, 29, 30, 31, 32, 35, 36, 37, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 60, 62, 63, 64, 65, 66, 67, 72, 73.	1, 3, 4, 5, 11, 21, 24, 31, 32, 37, 42, 44, 46, 47, 50, 53, 54, 55, 62, 63, 65, 66, 67, 73
44	24	Open Access	Data sets can be used, re-used and distributed without any restrictions.	The easier the use of datasets the higher the levels of transparency that can be created.	1, 3, 4, 5, 9, 10, 11, 20, 21, 23, 24, 25, 27, 29, 30, 31, 32, 35, 36, 37, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 60, 62, 63, 64, 65, 66, 67, 72, 73.	1, 3, 4, 5, 11, 21, 24, 31, 32, 37, 42, 44, 46, 47, 50, 53, 54, 55, 62, 63, 65, 66, 67, 73.
27	10	Ontology	Definitions and relationships between data elements to ease usage by machines and the understanding of users.	The better datasets are described the higher the level of transparency.	3, 4, 8, 15, 27, 28, 29, 30, 33, 34, 35, 44, 48, 49, 51, 52, 53, 54, 56, 60, 63, 64, 65, 66, 68, 70.	3, 4, 33, 44, 53, 54, 63, 65, 66, 68.
27	16	Timeliness	Data sets are up-to-date.	Up-to-date data is a condition for transparency.	3, 4, 14, 16, 25, 28, 30 ,31, 35, 38, 44, 47, 51, 52, 53, 55, 57, 59, 60, 61, 62, 63, 66, 71, 72, 73.	3, 4, 14, 16, 31, 44, 47, 53, 55, 59, 61, 62, 63, 66, 71, 73.
26	13	Primary	Data sets are disclosed in non-aggregated form or are preselected.	Access to raw data is a condition for transparency.	4, 10, 14, 18, 21, 22, 26, 28 ,29, 30, 35, 39, 44, 45, 46, 48, 51, 52, 54, 57, 59, 63, 64, 71, 73.	4, 14, 21, 22, 26, 39, 44, 46, 54, 59, 63, 71, 73.
24	14	Accuracy	The correctness of datasets.	More accurate data will result in higher levels of transparency.	4, 8, 11, 15, 27, 29, 38, 39, 42, 44, 45, 47, 48, 52, 53, 54, 56, 59, 61, 63, 71, 72, 73.	4, 11, 38, 39, 42, 44, 47, 53, 54, 59, 61, 63, 71, 73
20	11	Trusted	The conviction among users that data sets are reliable and accurate.	Higher levels of trust in data will result in the use of data for creating transparency.	8, 17, 18, 26, 32, 38, 40, 44, 45, 47, 51, 52, 53, 55, 59, 60, 61, 71, 72, 73.	26, 32, 38, 40, 44, 47, 55, 59, 61, 71, 73.
20	13	Web 2.0 Features	Inclusion of features enabled by the change from static websites to dynamic (e.g. videos, audio, animations) and user content generation (e.g. social media such as Facebook and Twitter).	Web 2.0 features will make the use of data for transparency more attractive.	16, 17, 19, 20, 22, 23, 30, 37, 45, 50, 55, 57, 59, 61, 62, 63, 66, 67, 72, 73.	16, 19, 22, 37, 50, 55, 59, 61, 62, 63, 66, 67, 73.
19	14	Openness appetite	Governmental level of desire to disclose and structure OGD.	The more data is opened the higher the level of transparency.	37, 39, 44, 45, 47, 48, 50, 55, 56, 57, 61, 62, 63, 64, 66, 67, 68, 69, 72.	37, 39, 44, 47, 50, 55, 61, 62, 63, 66, 67,68, 69, 72.
17	13	Technical capacity	Having the expertise and skills to disclose data for creating transparency.	Expertise and skills to disclose datasets is a condition for releasing data for creating transparency.	33, 37, 38, 39, 42, 47, 49, 54, 55, 56, 57, 59, 61, 62, 66, 67, 72.	33, 37, 38, 39, 42, 47, 54, 55, 59, 61, 62, 66, 67.
15	5	Machine-processable	Data sets are structured in such a way that they can be read and processed by machines.	Automatically processable data will result in higher levels of transparency.	15, 21, 27, 28, 30, 45, 48, 50, 51, 53, 57, 59, 60, 64, 65.	21, 50, 53, 59, 65.
15	6	Open data legislation and plan	A set of legislation and plans to create, maintain and update the OGD.	The more openness is enabled by legislation the higher the level of transparency.	2, 3, 10, 31, 43, 45, 49, 57, 62, 64, 66, 67, 68, 70.	3, 31, 62, 66, 67, 68.
14	7	Usability	The degree to which OGD portals are able to be used or are fit for use by citizens. Also described as user experience and user interface needs.	Higher usability will result in higher levels of transparency.	4, 9, 11, 14, 27, 29, 37, 45, 49, 52, 62, 63, 72, 73.	4, 11, 14, 37, 62, 63, 73.
14	4	Linked data	Method that structures data to be interconnectable.	Linking datasets will result in higher levels of transparency.	28, 30, 35, 44, 45, 48, 49, 51, 53, 60, 63, 64, 65, 70.	44, 53, 63, 65.
13	9	Complete	Data sets should contain all elements.	Complete datasets will result in higher levels of transparency.	14, 22, 26, 27, 29, 39, 45, 46, 59, 63, 64, 71, 73.	14, 22, 26, 39, 46, 59, 63, 71, 73.
13	8	Compliance	Adherence of systems to standards.	Use of standards will result in higher levels of transparency.	2, 3, 31, 43, 52, 54, 57, 62, 64, 67, 68, 69, 71.	3, 31, 54, 62, 67, 68, 69, 71.
13	4	Metadata	Metadata are the data that explain the meaning of the data within a dataset.	Datasets described using metadata will result in higher levels of transparency.	17, 28, 29, 48, 49, 51, 54, 56, 60, 64, 65, 71, 73.	54, 65, 71, 73.
13	5	Secrecy	Not disclosing information.	Keeping data secret will block transparency.	6, 12, 27, 29, 36, 38, 43, 46, 47, 48, 54, 70, 71.	38, 46, 47, 54, 71.
12	6	Centralized and permanent history	Having a centralized, single OGD portal with permanent history to ensure that data from the past can still be accessed.	Access to historical data and experiences with data will result in higher levels of transparency.	33, 35, 39, 47, 48, 51, 54, 57, 59, 60, 64, 66.	33, 39, 47, 54, 59, 66.
9	5	Searchability	A feature that allows citizens to find what they are looking for.	The ability to search for datasets in portals will result in higher levels of transparency.	4, 9, 12, 19, 21, 28, 37, 49, 73.	4, 19, 21, 37, 73.
8	1	Nonproprietary	Data formats are not the property of any institution (as this might limit use).	Using nonproprietary data will result in higher levels of transparency.	28, 45, 48, 51, 52, 53, 57, 60.	53.
7	1	Internal resistance	Bureaucratic and political reluctance to open data.	Higher levels of resistance will result in releasing less data for creating transparency.	9, 10, 36, 43, 70, 71, 72.	71.
7	6	Lack of regulatory Framework	Legislation guiding or requiring the disclosing of data.	Lack of legislation for opening data will result in less datasets for creating transparency.	56, 62, 63, 67, 68, 69, 71.	62, 63, 67, 68, 69, 71.
7	2	Misinterpretation	Having a wrong understanding of the meaning of the data.	Inability to interpret or wrong interpretation of data will result in lower levels of transparency.	7, 17, 18, 19, 40, 56, 57.	7, 40.
7	5	Unbiased or flawed methodology	Biased sensors or flawed statistical methodologies providing incorrect data or outcomes.	Biased and flawed data will result in lower levels of transparency.	4, 47, 52, 54, 61, 68, 72.	4, 47, 54, 61, 68.
6	6	Lack of supervision	Management supervision over the handling of disclosure to ensure opening of data.	A lack of management supervision when opening data will result in lower levels of transparency.	62, 63, 67, 68, 69, 71.	62, 63, 67, 68, 69, 71.
6	4	Language Solutions	Supporting different levels of language proficiency or using ideograms to ease understanding.	Language support will result in higher levels of transparency.	29, 31, 35, 55, 65, 67.	31, 55, 65, 67.
6	2	Public Input	Different types of access to support a variety of purposes.	Providing different ways to access data will result in higher levels of transparency.	13, 20, 29, 51, 53, 60.	13, 53.
5	4	Government budget	Bigger budgets allow more opportunity to create, maintain and update an OGD public policy.	Higher budgets for opening data will result in higher levels of transparency.	25, 33, 66, 67, 68.	33, 66, 67, 68.
5	3	Information overload	Offering much more data than people can process.	Opening non-relevant data will result in lower levels of transparency.	7, 18, 19, 28, 40.	7, 19, 40.
5	3	Party policy preference	The political preferences regarding OGD data policy.	Ruling politicians favoring open data will result in higher levels of transparency.	46, 56, 63, 64, 66.	46, 63, 66.
5	3	Security	Data should be protected from manipulation.	Higher levels of security will result in higher levels of transparency.	3, 51, 53, 59, 60.	3, 53, 59.
4	2	FAQs and examples	Frequently asked questions about recurring problems and examples to illustrate good practices.	Support will result in higher levels of transparency.	3, 36, 59, 64.	3, 59.
4	1	License-Free	No license restricting open data use.	Non-restricting licenses for using datasets will result in higher levels of transparency.	51, 52, 53, 60.	53.
4	3	Size	The number of employees of a public organization.	The larger the public organization the more datasets will be released and the higher the level of transparency.	25, 37, 67, 68.	37, 67, 68.
3	3	Political pressure	Influence of other countries on initiating OGD.	Political pressure from other countries will result in higher levels of transparency.	32, 33, 67.	32, 33, 67.
3	1	Opacity	Disclosing data in such a way that it cannot be used.	Opacity will result in lower levels of transparency.	46, 49, 60.	46.
2	1	Intermediaries	Intermediaries processing open data and making them available using software applications and websites.	The more intermediaries who are involved in processing open data the higher the levels of transparency.	45, 68.	68.
2	2	Organizational decentralization	The level of (de)centralisation of decision-making.	Centrally organized organizations will release more data, resulting in higher levels of transparency.	32, 67.	32, 67.
2	0	Technology literacy	The ability to understand and use OGD.	The more literate citizens are, the higher the levels of transparency.	9, 29.	–
2	1	Type of organization	Governments are diverse and range from small to large. Also activities might be outsourced to private parties, who have no obligation to create transparency.	The more complex the administration is, the lower the level of transparency.	32, 56.	32
2	2	Urban area	Geographical location of a government.	Governments in densely populated areas will create higher levels of transparency.	33, 69.	33, 69

Additionally, discussions can be found in the literature about what happens when inaccurate data are shared, and the issue of who is accountable for decisions based on inaccurate data (French, 2011). Finally, some work mentions both the positive and negative effects of transparency. Appendix A – The overall classification of the papers summarizes the descriptions and the types of effects identified in the SLR of transparency.

The most mentioned effects of transparency are headed by accountability (51 papers), followed by trust and credibility (35 papers), civic engagement and participation (30 papers), efficiency and reduction of costs (28 papers), governance and political turnout (20 papers), anti-corruption efforts (17 papers), error and asymmetry of information (11 papers) and privacy (6 papers). The papers often mention more than one expected effect. As an example, Janssen, Matheus, and Zuiderwijk (2015) found that transparency can increase accountability and can help anti-corruption efforts. That an effect is mentioned does not mean that the effect is also achieved. There is still discussion about whether transparency results in accountability (Park & Gil-Garcia, 2017).

Most of the literature is focused on the positive effects of transparency. The number of papers mentioning negative effects is low: we found that only 16% of the papers (12 out of 73) mentioned negative effects. The average of expected effects is in the surveyed PA literature 62% higher than in IS. In PA there is a wider discussion of the effects of transparency, whereas in IS literature transparency is often viewed as the outcome (dependent variable). Not surprisingly, discussions in IS are more focused on technical solutions and the determinants of transparency.

The window theory

The disclosure of OGD should result in transparency by creating a window on governmental functioning. Our aim is to develop a comprehensive model of possible determinants that enable or impede transparency for OGD and the efficacy of transparency. Hence, the Window Theory is not a tested theory, but a comprehensive account of the determinants and effects of transparency. We label it as a theory as it is a supposition of factors aimed at explaining how transparency can be created and what its effects are. The SLR provides a long list of factors but gives little insight into which factors are conditions, which would be good to include, which are nice to have and which have hardly any influence. Nor does literature give insight into the significance of factors or whether they are antecedents or moderators. Therefore, we opted for creating a comprehensive model that contains all determinants deemed to be relevant, as shown in Figure 2. The model shows the relationship between 42 determinants and 8 possible effects and unifies various views on transparency within a single model. Some relationships are likely to be strong, whereas others will be weak. The determinants are grouped into 4 clusters and the effects into 1 cluster based on the content analysis.

The resulting model is named “The Window Theory” as OGD should provide a window to see what has happened or what is happening within the government. The frame of the window determines if it can be used. This is expressed by the “organizational characteristics” and “system quality” cluster. In the window, the “data quality” cluster determines if the data becomes transparent. Governments are creating the window to the outside world by disclosing OGD in portals. The determinants show that simply disclosing OGD is not sufficient and that a large number of determinants need to be addressed. Addressing the determinants can governments to arrive at OGD portals that really create transparency.

As a metaphor: a window is very effective during the day, but at night, nothing can be viewed without artificial light. The model shows that the night will remain dark unless users are guided towards the data. The same data that create a greater level of transparency for the expert, offer less transparency for someone with different conditions of access and a lack of knowledge about how to use the data. This aspect is addressed by the “individual characteristics” cluster.

The many determinants and its relationship suggest that transparency should be viewed as a complex, continuous construct rather than a dichotomous construct. Transparency is hard to express on a single scale ranging from zero to full transparency. Furthermore, the two extremes (zero or full transparency) are less likely to occur as there will always be some degree of transparency.

There are four main clusters of determinants identified, as shown in Figure 2. In the figure in parentheses is the total count of the individual times that the groups of factors are mentioned.

• 1.  Data quality (315): The aspects determining the quality of the OGD published

• 2.  System Quality (56): Characteristics of systems that are used for publishing data

• 3.  Organizational characteristics (135): Variety of organizations that publish the OGD

• 4.  Individual characteristics (21): Variety of users of the OGD

Data quality determinants were found most frequently. In almost all the IS papers data quality is mentioned. One of the explanations for this is that data quality plays a major role in the Delone and McLean (2003) success model for information systems, one of the most cited papers in the IS field. This model also includes system quality dimensions, but this is less evident in the literature surveyed. In most IS literature system quality is not viewed as being directly connected to transparency, and in the first three phases (Figure 3), system quality factors are rarely mentioned in the literature. Only recent literature, mostly after 2014, acknowledges that system quality also influences transparency. The simple release of data was not found to be sufficient, and characteristics such as usability, performance (for real-time data provision) and comparability are found to be important for creating transparency (Bertot et al., 2014; Janssen & Kuk, 2015; Muñoz-Soro, Esteban, Corcho, & Serón, 2016).

Both organizational and individual characteristics play a role, according to the literature. Organizational characteristics are diverse and range from leadership to geographic (urban) area. Some organizations are better prepared and are better equipped to create transparency, while others lack these capabilities (Barry & Bannister, 2014; Guillamón, Ríos, Gesuele, & Metallo, 2016; Ølnes, 2016). Political orientation also plays a role, as some political movements favor disclosure and transparency, whereas others do not (Gandía et al., 2016; Reynaers & Grimmelikhuijsen, 2015).

Users can be diverse, possessing varying individual characteristics, as depicted in Figure 2. Users can be citizens (Bertot et al., 2010), company employees (David‐Barrett & Okamura, 2016), public servants from other organizations (Angst, Agarwal, Gao, Khuntia, & McCullough, 2014), persons working in public-private organizations delivering public services (Reynaers & Grimmelikhuijsen, 2015), and employees of supra-national organizations (Cross, 2014). User characteristics such as technology literacy (Owen, Cooke, & Matthews, 2013), level of education (Grimmelikhuijsen & Meijer, 2014), and gender (Bauhr & Grimes, 2014) influence the ability to make use of data. Some users only read news from intermediaries such as journalists, who themselves used OGD and FOI (Piotrowski & Van Ryzin, 2007), or use applications providing a predefined view on the data.

The Window Theory contains a long list of factors, but not all factors are always relevant for every situation at hand. Our analyses suggest that the determinants are dependent on the desired effects of transparency and the context. Dependent on what the aim of transparency is, other factors were deemed to be found relevant in the literature. Metaphorically speaking, different windows are needed for different purposes. This implies that for a given situation the Window theory can be used as the basis, but determinants and factors need to be selected to make them context-specific. This can also explain the many different models and lack of uniformity in the literature review. Dependent on the context, user group and objective different aspects are of importance.

Discussion and further research

The Window Theory is based on the idea that OGD creates a window through which to observe government, and thus to overcome information asymmetry. There are two parties, those who look through the window and those who determine what is showcased in the window. OGD is showcased by governments. What is showcased reflects the intentions of the publisher, but might not fit the needs and desires of what the public wants to see. Glass is often set in a window to protect against weather conditions, but blurs the view. If not properly done there can be dust on the window. The physical characteristics of glass let it transmit, reflect and refract light. In a similar vein, many transparency efforts only reveal a part of the full picture, reflecting what you want to see and not necessarily resulting in transparency. The quality of glass can be enhanced by cutting and polishing. Policy-makers and designers can use a comprehensive model to enhance transparency by addressing the determinants.

Transparency is a multidimensional concept (Furtado et al., 2010), which makes it both hard to define and to create. Opening too much data without proper guiding mechanisms can only result in blurring the window and actually preventing the viewer from seeing what is happening inside the government. This is labeled as “disclosure overload” in compliance and regulation literature (Hoitash & Hoitash, 2017). In this field, regulators ask for more and more information and reporting, which in turn is hardly used by the agencies in charge of the supervision. The volume of disclosure is not helpful in creating transparency and might even have the opposite effect. Removing the blur and presenting the public with easily understandable information presented in a manageable form requires significant investment. The level of transparency is influenced by the type of window (data, quality) but also by who is looking (stakeholders), from where (context, position in charge), when (time) and how people look through the same window (objective use of data). Research is needed on how to lower the threshold of public involvement and to avoid disclosure overload.

The most comprehensive model addressing transparency that we identified using SLR defined 16 determinants and a maximum of 3 effects. Our model is focused on transparency enabled by OGD and is the most comprehensive, as the Window Theory lists 42 determinants and 8 expected effects. The disadvantages of our model are related to the many factors involved. Furthermore, the relative importance of each factor is unclear. The number of works on the subject of data quality and transparency was found to be high (+315), however research is not evenly distributed across all factors. For example, most quantitative transparency models do not take the technical aspects into account, whereas case-based research shows that these can be highly relevant. In addition the relative importance found in one domain might not hold in the OGD domain. We recommend conducting empirical research into the relative importance of factors.

The findings show that the information systems literature focuses more on determinants such as information quality and system quality dimensions, whereas public administration research focuses more on aspects such as trust and accountability. Give the focus of the domains this comes not as a surprise. However, these fields seem to be converging, a development that is expressed in current work that attempts to bridge the gap between them (Bertot et al., 2014; Janssen & Kuk, 2015; Muñoz-Soro et al., 2016). The work combining elements from the public administration and information systems field might be best characterized as the “digital government” domain. It is likely that these domains will learn from each other and will start to influence each other. More research is needed in order to understand the influence and relevance of the factors involved, in particular whether a determinant is a condition, a facilitator or an inhibitor of transparency. The SLR shows that the importance of determinants is dependent on the intended effect. Also, the significance of factors might be dependent on the type of OGD and the policy and institutional context. We excluded literature concerning budgetary data as this is well-structured and well-defined and guided by regulations, whereas this is not the case for most OGD. A comparison between OGD and budgetary data can be made in further research. We recommend using the Window Theory in future research as a starting point for identifying most influential factors and developing a parsimonious model. Also, we recommend the development of contextual models that provide insight into the deeper structure and meaning of determinants and effects.

The complexity and the quantity of determinants challenge the assertion that the opening of data will result in transparency. The simple opening of data is unlikely to result in transparency per se. There is no single recipe for creating transparency. Some determinants might have a larger influence than others and some might be conditions for creating transparency, but the literature does not provide a clear view on this. Our review suggests that transparency is context-dependent and that the creation of transparency depends on what information is supplied, how it is provided and on its users, who need to have the knowledge and the capabilities to understand the open data. Alongside the data, details of the context have to be shared in order to allow the public the opportunity to interpret the data.

One of the main ideas behind OGD is to create transparency for the general public. The determinants show that this might be more complicated than initially expected. Individual characteristics hinder the use of OGD to create transparency. Hence, the creation of mass-transparency is not easy to accomplish. The requisite individual characteristics are only found in a relatively small group, and consequently, transparency is only created for the happy few. By taking the comprehensive list of determinants into account, the Window Theory can be used to facilitate a much broader audience.

Budget transparency was left out in our SLR, as this is a well-defined domain with a long track record in the standardization of financial reporting. It is likely that a subset of determinants and effects are relevant to budget transparency. Further research can evaluate whether all factors for budget transparency can be found in the Window Theory and whether a specific subset can be derived for this domain.

The Window Theory started with transparency and investigated both determinants and effects. Use is a central construct influencing transparency. Without the use of OGD, there can be no creation of transparency. Use is a complicated construct and often requires participation and interaction amongst people. Citizen’s engagement and participation were also found to result from transparency (Harrison & Sayogo, 2014). Although both elements are related to engagement and participation, engagement and interaction are needed as determinants to make sense of the OGD. Once OGD is understood and transparency is created this can result in engagement and participation in order to influence policy-making and political decisions. As such, these two elements should be separated and not combined. Making sense of OGD by interaction and using analytical capabilities is an area of limited research. We only found a few studies that referred to this aspect, whereas it might be the most important element in creating transparency once OGD is published in a suitable format.

Transparency covers a broad range of concepts and elements. In the SLR we encountered the fact that in the literature different names are used to express similar determinants and effects, although there are sometimes slightly different interpretations. This complicated the aggregating of the determinants and effects and the comparison. For example, completeness and incompleteness refer to a variable showing various degrees of completeness, ranging from zero data to complete data. Another example of using different names for what could be similar concepts is the use of the words “trust” and “reliability” of data. Both can refer to data that are defined clearly enough to arrive at similar results, but reliability can refer to being continuously available and trust can refer to whether the data provider can be trusted. Trust is often viewed as being broader than reliability, with reliability as one of its properties. The use of the terms seem to be dependent on the community: PA prefers to use trust, while IS typically prefers to use the term reliability. A limitation of our work is that we merged some of the concepts into one to keep the number of determinants manageable. Although the determinants provide a solid starting point, their descriptions in Table 2 should be refined in further research.

The literature suggests that full transparency is hard to achieve, but is also unnecessary (Roberts, 2002). Given the many factors involved, it seems to be better to discuss the level of transparency. However, the question remains of how much transparency is sufficient, and this might be dependent on the situation at hand and the users. We do not need to scrutinize public servants at all times during their work, and the privacy of public servants should be respected. A level of trust is necessary, that the public servant is able to do his job properly and the prevailing culture should ensure that public services are executed in a professional way.

Politicians want to create transparency using OGD, whereas public servants still need to realize this. The Window Theory shows that there is a large discrepancy between the intuitive notion and the actual realization. Many open data portals release OGD, but this does not always create transparency, as determinants of transparency have not been considered or not been handled in a suitable way. It is too simple to assert that transparency is automatically created by realizing OGD. We suggest to use the determinants as a form of checklist to evaluate transparency initiatives and to use this evaluation to broaden the view on creating transparency. Also, the impact of creating transparency can be more diverse, as the effects show.

Conclusion

The creation of transparency using OGD has become one of the key areas in digital government research. The literature review showed an increase in papers related to open data and transparency over time, but also fragmentation, as authors look at different determinants and effects of transparency. There was no comprehensive overview, which makes it hard to determine which factors should be deemed to be important. Many studies only take a few actors into account.

We unraveled the concept of transparency for OGD by finding 42 determinants influencing transparency and 8 types of expected effects. The determinants were clustered in four groups of factors, data quality, system quality, organizational characteristics, and individual characteristics.

The Window Theory is an unifying model containing these determinants and effects of transparency. By creating an overview, the Window Theory unifies the various work in this area originating from public administration and information system publications. The nature and objectives of transparency are likely to differ per situation and the model can be used to select the determinants that are appropriate to a given context. Factors might be more or less important dependent on the situation.

Our SLR depends on previously published research and the availability of these studies, which limits its outcome. We had to interpret the descriptions in the literature which were often ill-defined and ambiguous and sometimes suggested relationships while lacking a clear description of the explanatory mechanisms. We were not able to discriminate between antecedents and moderators. Empirical research is needed to develop explanations of the relationship between the determinants of and effects of transparency. In further research, a distinction between factors as a condition, facilitator or inhibitor of transparency should be made. The determinants with the highest explanative power can be selected to develop a parsimonious model. Furthermore, we recommend further research to focus more on what data is needed and in what form, and on the interpretation and sense-making of data.

Additional information

Funding

Part of this work is funded by the European Commission within the H2020 Program in the context of the project OpenGovIntelligence (www.opengovintelligence.eu) under grant agreement [No. 693849].

Notes on contributors

Ricardo Matheus

Ricardo Matheus is a researcher in the field of Open government Data and infrastructures at the Information and Communication Technology research group of the Technology, Policy and Management Faculty of Delft University of Technology (The Netherlands). He was a lecturer at Rotterdam School of Management of Erasmus Rotterdam University (The Netherlands) teaching Data Science and Programming for Managers courses. He leads WPs in the CAP4CITY Project (www.cap4city.eu/) and managed WPs in the OpenGovIntelligence project (www.opengovintelligence.eu) which was aimed at creating transparency using open government data in six governmental pilots.

Marijn Janssen

Prof. Dr. Marijn Janssen is a full Professor in ICT & Governance and head of the Information and Communication Technology research group of the Technology, Policy and Management Faculty of Delft University of Technology. His research interests are in the field of orchestration, shared services arrangements, and open and big data. He is Co-Editor-in-Chief of Government Information Quarterly, conference chair of IFIP EGOV series and is chairing mini-tracks at e-government and information systems conferences. He has published over 500 refereed publications. He was nominated in 2018 and 2019 by Apolitical as one of the 100 most influential people in the Digital Government worldwide https://apolitical.co/lists/digital-government-world100. More information: www.tbm.tudelft.nl/marijnj.

Appendix A: The overall classification of the papers

Table

ID	Reference	Area	Year	ID	Reference	Area	Year
1	Piotrowski and Van Ryzin (2007)	PA	2007	38	Grimmelikhuijsen and Meijer (2014)	PA	2014
2	Blomgren (2007)	PA	2007	39	Hollyer, Rosendorff, and Vreeland (2014)	PA	2014
3	Pina, Torres, and Royo (2007)	PA	2007	40	de Fine Licht (2014)	PA	2014
4	Rawlins (2008)	PA	2008	41	De Fine Licht, Naurin, Esaiasson, and Gilljam (2014)	PA	2014
5	Relly and Sabharwal (2009)	IS	2009	42	Bauhr and Grimes (2014)	PA	2014
6	Coglianese (2009)	PA	2009	43	Gregory Michener (2015)	PA	2014
7	S. Grimmelikhuijsen (2009)	PA	2009	44	Tan (2014)	PA	2014
8	Furtado et al. (2010)	IS	2010	45	Zyl (2014)	PA	2014
9	Bertot et al. (2010)	IS	2010	46	Cross (2014)	PA	2014
10	Jaeger and Bertot (2010)	IS	2010	47	Auger (2014)	PA	2014
11	McNeal and Hale (2010)	PA	2010	48	Bertot et al. (2014)	PA	2014
12	Worthy (2010)	PA	2010	49	Barry and Bannister (2014)	PA	2014
13	French (2011)	PA	2011	50	Ganapati and Reddick (2014)	PA	2014
14	Grimmelikhuijsen (2011)	PA	2011	51	Janssen and van den Hoven (2015)	IS	2015
15	Chen (2012)	IS	2012	52	Lourenço (2015)	IS	2015
16	Bonsón, Torres, Royo, and Flores (2012)	IS	2012	53	Abu-Shanab (2015)	IS	2015
17	Evans and Campos (2013)	PA	2012	54	Kashin, King, and Soneji (2015)	PA	2015
18	Grimmelikhuijsen (2012)	PA	2012	55	Porumbescu (2015)	PA	2015
19	Bridges, Appel, and Grossklags (2012)	PA	2012	56	Reynaers and Grimmelikhuijsen (2015)	PA	2015
20	Chun and Cho (2012)	PA	2012	57	Worthy (2015)	PA	2015
21	Munson et al. (2012)	PA	2012	58	Grimmelikhuijsen and Klijn (2015)	PA	2015
22	Van Ryzin and Lavena (2013)	PA	2013	59	Venkatesh, Thong, Chan, and Hu (2016)	IS	2016
23	Meijer (2013)	PA	2013	60	Shaikh and Vaast (2016)	IS	2016
24	Themudo (2013)	PA	2013	61	Chen, Vogel, and Wang (2016)	IS	2016
25	Rodríguez Bolívar, Alcaide Muñoz, and López Hernández (2013)	PA	2013	62	Navarro-Galera et al. (2016)	IS	2016
26	Deng, Peng, and Wang (2013)	PA	2013	63	Gandía et al. (2016)	IS	2016
27	Greg Michener and Bersch (2013)	PA	2013	64	Muñoz-Soro et al. (2016)	IS	2016
28	Lourenço (2013)	PA	2013	65	Höffner, Martin, and Lehmann (2016)	IS	2016
29	Owen et al. (2013)	PA	2013	66	Guillamón et al. (2016)	IS	2016
30	Halachmi and Greiling (2013)	PA	2013	67	Grimmelikhuijsen and Feeney (2016)	PA	2016
31	Harrison and Sayogo (2014)	IS	2014	68	David‐Barrett and Okamura (2016)	PA	2016
32	Cuadrado-Ballesteros (2014)	IS	2014	69	Peiffer and Alvarez (2016)	PA	2016
33	Angst et al. (2014)	IS	2014	70	Graham, Gooden, and Martin (2016)	PA	2016
34	Packer, Diochnos, Rovatsos, Gal, and Moreau (2014)	IS	2014	71	Ríos, Bastida, and Benito (2016)	PA	2016
35	Martin, Stadler, Frischmuth, and Lehmann (2014)	IS	2014	72	Reddick, Chatfield, and Brajawidagda (2016)	PA	2016
36	Villeneuve (2014)	IS	2014	73	Song and Lee (2016)	PA	2016
37	Yavuz and Welch (2014)	IS	2014