Improving the quality of environmental impacts assessment reports: effectiveness of alternatives analysis and public involvement in JICA supported projects

Abstract

This study examined the key factors for improving the quality of environmental impact assessment reports through statistical tests and path analysis. The Lee-Colley review package was used to review the quality of the samples of 120 reports prepared by the Japan International Cooperation Agency dating from 2001 to 2012. The result of the study showed that alternatives analysis and public involvement could be key factors for improving the quality of reports. When the number of public involvement stages went up, the number of evaluation criteria for alternatives analysis showed an increasing trend and the quality of the reports improved. Finally, the study pointed out the effectiveness of alternatives analysis with a wide range of evaluation criteria and public involvement for improving the quality of reports. Further research is needed to explore alternatives analysis and public involvement in more depth as well as to improve the effectiveness of their linkage via more case studies.

Keywords:

• Alternatives analysis
• public involvement
• statistical tests
• path analysis
• the Lee-Colley review package

Introduction

The quality of the environmental impact assessment reports (EIARs) is one of the three major dimensions of an effective environmental impact assessment (EIA) system, which consists of: (1) adequate institutional arrangements for EIA; (2) the quality of EIARs; and (3) implementation of mitigation measures (Ortolano et al. 1987; Sadler 1996; and Momtaz & Kabir 2013). The quality of EIARs is very important for the effectiveness of the EIA system and for making good decisions. However, the results of the review of the EIARs submitted to the Asian Development Bank revealed that they were generally weak in: (1) assessment of ecological impacts; (2) analysis of alternatives; (3) economic analysis of environmental impacts; and (4) public participation (Lohani et al. 1997, p. 2–31). Almost 10 years have passed since the above-mentioned finding, yet weak enforcement has still been a major problem in many developing countries in East and Southeast Asia, which was reflected through late implementation, insufficient consideration of alternatives, weak public consultation, and a lack of information disclosure. In order to make the system more effective, the requirements of early implementation, alternatives analysis, public consultation, and information disclosure should be stipulated as essential for EIA (World Bank 2006, p. 15).

The Japan International Cooperation Agency (JICA), which assists and supports developing countries as the executing agency of Japan’s official development assistance (ODA), began applying guidelines for environmental and social consideration (ESC) in April 2004. The JICA ESC guidelines institutionalized procedures for EIA at the preparation phase of the project cycle. These included such things as screening classifying projects into three categories, assessing a wide range of environmental and social impacts, analyzing alternatives including zero option, introducing strategic environmental assessment (SEA), information disclosure, and public involvement. In 2010, JICA fully widened the range of the EIA process to the project cycle from the preparation phase to monitoring phase (JICA 2010). Before introduction of JICA ESC guidelines in 2004, the JICA had environmental guidelines for infrastructure projects (prepared in 1990). The 1990 guidelines introduced screening and scoping processes into the feasibility study that were based on the Recommendation of the Council on environmental assessment of development assistance projects and programs of the Organization for Economic Co-operation and Development (OECD) in 1985 (OECD 1985). The JICA guidelines in 1990 did not include regulations relating to alternatives analysis, public involvement, impact prediction, impact significance, mitigation, or monitoring. Those processes were applied on a voluntary basis and the implementation was insufficient.

When the guidelines in 2004 were introduced, JICA established the advisory committee for ESC as a permanent third-party institution formed by external experts. This committee is a system unique to JICA, which seeks external advice on ESC in the decision-making processes of projects. The committee’s meetings are held in a public forum. The advice given at the meetings is made public, therefore boosting the transparency and credibility of JICA’s decision-making process. In the two years from July 2010, the JICA ESC committee provided 1,123 advices in total for 40 projects, The number of advice about alternatives and public involvement was 75 and 131, respectively (Murayama et al. 2012). The alternatives analysis and public involvement were big issues for JICA’s SEA and further efforts were needed to improve both of them (Masumoto et al. 2013). The alternative analysis and public involvement are tasks for improving the quality of JICA EIARs in the future.

The identification and comparison of alternatives are central to the application of EIA as a creative, problem-solving process. However, review of the alternatives is poorly represented or inadequately carried out in many countries. The important role of public involvement in EIA is a reflection of the wider sociopolitical context, and the successful and consistent integration of public involvement and consultation with development-related decision-making have still not been achieved (Abaza et al. 2004, p. 51 and 65). In spite of its importance, the little progress of alternative considerations has been observed over the years. The poor consideration of alternatives is exacerbated by the separation of the impact assessment types. The better integration of types is necessary to ensure that impact assessment adds more value to decision-making (Geneletti 2014, p. 17). The benefits of public participation are: improved quality of decisions, minimizing cost and delay, consensus building, increased ease of implementation, avoiding worst-case confrontations, maintaining credibility and legitimacy, anticipating public consensus and attitudes, and developing civil society. Until recently, the benefits of public participation in EIA are indicated empirically and the empirical research is limited (Creighton 2005, p. 19). Okello et al. (2009) indicated that despite good regulations regarding public participation in Kenya, practice was poor particularly during the scoping, report review, and follow-up stages. Nadeem and Fischer (2011) evaluated the performance of public participation in EIA in Pakistan and revealed an overall weak influence of public participation on the substantive quality of EIA and on the final decision. Despite the importance of public involvement, its performance is not high.

There are many researches of reviewing the quality of EIARs (Wood et al. 1996; Lee 2000; Cashmore et al. 2002; Canelas et al. 2005; Tzoumis 2007; Sandham & Pretorius 2008; Badr et al. 2011; Momtaz & Kabir 2013; Sandham et al. 2013; and Chanthy & Grünbühel 2015). Previous studies used the summary tables to illustrate grades and the portion of grades that were satisfactory and unsatisfactory, and then went on to discuss the improvement in quality. Findings from these studies indicated the description of the project and the environment, and communication of results tended to be the better performed areas, whereas impact identification, alternatives and mitigation tended to be less well performed. Many constraints on quality of EIARs were identified in developing countries including limited published research, EIA practitioners not being independent from developers, inadequate baseline data, lack of EIA experts, and a shortage of study time. However, little is known about the countermeasures for improving the quality of EIARs under the above-mentioned constraints. Alternatives analysis and public involvement could be key factors for improving the quality of EIARs but there was very little evidence to show a causal relationship between the alternatives analysis and public involvement, and the quality of EIARs. A statistical analysis for grade data may help to identify the key factors that influenced the quality of EIARs and to propose concrete measures for improvement. Statistical analysis has the advantage of distinguishing whether the difference between the groups is the effect of a specific factor or merely a coincidence. In addition to this, the use if statistical analysis means that a large number of samples can be analyzed and findings obtained in a more general manner than that of the case study method. Therefore, using statistical analysis for grade data, this study aimed to find the key factors that influenced the quality of JICA EIARs and to propose countermeasures for improving the quality of the reports based on the results of the statistical analysis. For the purpose of this study, samples from 2001 to 2012 were used because it was expected to improve the quality of JICA EIARs from the introduction of the JICA ESC guidelines in 2004 and to find key factors influencing quality by analyzing grade data before and after 2004. The research results may be useful for improving not only the quality of JICA EIARs but also that of EIARs in developing countries.

Methods

Selection of sample

The study reviewed the quality of 120 JICA EIARs from 2001 to 2012, to see whether the introduction of JICA ESC guidelines in 2004 had any effect on quality, and to identify the key factors influencing the improvement of quality. A total of 120 samples – 10 per year for 12 years – were randomly selected using the Japanese Industrial Standards random number table. A list of reports searched for each year through the website of the JICA library was used as the population. The random samples are very likely to have the population statistics. The total number of EIARs was not given. The sample size was decided with reference to past case studies, which showed 112 samples were used in European countries (Wood et al. 1996), 72 in Greece (Cashmore et al. 2002), 50 in Estonia (Peterson 2010), 46 in Portugal and Spain (Canelas et al. 2005), 45 in Egypt (Badr et al. 2011), 39 in Cambodia (Chanthy & Grünbühel 2015), 30 in Bangladesh (Momtaz & Kabir 2013), and 28 in South Africa (Sandham & Pretorius 2008). The sample size exceeded the largest size of any previous study and was judged sufficient for the study.

Conducting a review of EIA report quality

In view of the widespread use and utility of review method, the quality review was based on the Lee-Colley review package (Lee et al. 1999), commencing at the lowest level. The review was conducted by a single reviewer, who held the qualification of professional EIA engineer in Japan. The package advised the two reviewers to control for subjective differences between individuals. The two reviewers needed to make an overall judgment on the acceptability of the EIAR as a planning document at the time of the environmental appraisal. A single reviewer assigned better assessment grades than a group of reviewers (Peterson 2010). Therefore, this study minimized the subjectivity of a single reviewer using a random sampling method, conducting a review of as many EIARs as possible by the same reviewer, and comparing grades between groups based on the design of experiments. According to this method, the quality review involves evaluating how well a number of assessment tasks, which are grouped hierarchically into sub-categories, categories and area, have been performed. The review areas and review categories, and the assessment symbols are summarized in Tables 1 and 2, respectively. Grades for higher levels of the hierarchy are not determined by a simple averaging of the assessments of the component, but by an overall performance grade per category and again for the review area. The JICA ESC guidelines cover social impacts in addition to the environmental impacts, but the Lee-Colley review package does not mention social impacts clearly so in this study, the environmental impacts were construed as including reference to social impacts. The category of wastes (1.3) was excluded from the review because it could be one part of the category of environmental description (1.4) and it did not need to be focused on.

Table 1. Review areas and review categories.

1	Description of the development, the local environment and the baseline conditions
	1.1	Description of the development
	1.2	Site description
	1.3	Wastes
	1.4	Environment description
	1.5	Baseline conditions
2	Identification and evaluation of key impacts
	2.1	Definition of impacts
	2.2	Identification of impacts
	2.3	Scoping
	2.4	Prediction of impact magnitude
	2.5	Assessment of impacts significance
3	Alternatives and mitigation
	3.1	Alternatives
	3.2	Scope and effectiveness of mitigation measures
	3.3	Commitment to mitigation
4	Communication of results
	4.1	Layout
	4.2	Presentation
	4.3	Emphasis
	4.4	Non-technical summary

Source: Lee et al. 1999.

Table 2. Assessment symbols.

Symbol	Explanation
A	Relevant tasks well performed, no important tasks left incomplete
B	Generally satisfactory and complete, only minor omissions and inadequacies
C	Can be considered just satisfactory despite omissions and/or inadequacies
D	Parts are well attempted but must, as a whole, be considered just unsatisfactory because of omissions or inadequacies
E	Not satisfactory, significant omissions or inadequacies
F	Very unsatisfactory, important tasks poorly done or not attempted
N/A	Not applicable. The review topic is not applicable or it is irrelevant in the context of the statement

Source: Lee et al. 1999.

Statistical test

The samples were divided into three periods of four-year intervals and the difference of grades was tested to see the introduction effect of JICA ESC guidelines in 2004. Sectors consisted of transportation, regional development, power, water resource, pollution control, and agriculture. The difference of grades between sectors was tested too. Alternatives analysis and public involvement are very important for the EIA process and were institutionalized in 2004. These two processes seemed to be key factors for improving the quality of JICA EIARs. In order to assess the effects of the relevance of alternatives analysis and public involvement on the quality of JICA EIARs, the 120 samples were stratified into four groups: (1) both processes of alternatives analysis and public involvement; (2) alternatives analysis process only; (3) public involvement process only; and (4) neither alternatives analysis nor public involvement process. The difference in grades between these four groups was tested.

Then, the samples were divided in two parts by presence or absence of alternatives analysis and each part was stratified into four groups by number of public involvement stages so as to assess the effect of public involvement on quality. Three times (PI3) means: public involvement at the scoping stage, the intermediate stage between the scoping and draft reporting, and the draft reporting stage; two times (PI2) refers to public involvement at the scoping stage and the draft reporting stage; one time (PI1) means public involvement at the draft reporting stage; and zero time (PI0) refers to no public involvement at any stage. The difference of grades between the four groups was tested in two parts. The next step was to assess the difference in effects between the number of public involvement stages, and the number of alternatives and evaluation criteria. The samples relating to presence of alternatives analysis (n = 76) were stratified into four groups by the number of public involvement stages. The difference in number of alternatives and evaluation criteria between the four groups was tested.

Statistical tests were performed using non-parametric tests (Kruskal–Wallis test or Spearman’s correlation coefficient by rank test depending on types of data) for an ordinal scales such as grades A to F of the Lee-Colley review package, and using the upper-sided Tukey-Kramer multiple comparison tests for ratio scales such as number of alternatives and evaluation criteria. The Kruskal–Wallis test was used to test the association between one nominal variable and one ranked variable. The Spearman’s correlation coefficient by rank test was used to test the association between two ranked variables. The Tukey-Kramer test was the most common multiple comparison used to test the difference between more than three groups. The difference with *p < .05 and **p < .01 was considered significant. Statistical tests were performed using Excel 2010, the add-in software Statcel3 (Yanai 2014), and the software multiple comparisons Toraneko (Ogura 2012).

Structural equation modeling

A path analysis with structural equation modeling (SEM) was conducted to obtain a causal model with path coefficients between five variables and the overall quality of JICA EIARs. This analysis could be used to show a causal relationship between related variables and quality, and to show the effect of each variable. The five variables were: number of alternatives, number of evaluation criteria, alternatives analysis, public involvement, and mitigation. The SEM is a statistical method and its use has rapidly spread in recent years. The method displays a statistical model with a path diagram that shows a causal relationship between variables with arrows in an easy-to-understand manner. It is very practical because observable variables and latent variables are used to analyze a causal relationship and it has been applied to social research and marketing methods. But it needs to set up a causal hypothesis using a path diagram in the first place. Moreover, the preparation of a hypothesis requires special knowledge of EIA. One big feature of path analysis with SEM is a set of adaptation indexes used to reject an incorrect model. Satisfactory adaptation indexes are necessary for a correct model. A causal model with satisfactory adaptation indexes could be evidence of the effects of variables on the quality of JICA EIARs. A correlation coefficient matrix relative to analysis target data is needed for path analysis with SEM. The ordinal scales from A to F were converted to rank scores like 6, 5, 4, 3, 2, and 1 to prepare the matrix. The grades of the review categories of alternatives (3.1) and the mitigation (3.2), and overall grades as well as the number of alternatives, evaluation criteria and public involvement stages were used as data for the path analysis with SEM. The path analysis with SEM was performed using Excel 2010 and the add-in software Covariance structure analysis (Kojima & Yamamoto 2013).

Results

Overall quality of EIAR sample

The analysis of the overall quality of JICA EIARs (Table 3) indicated that none of the reports could be described as having been well performed (A), 17 were generally satisfactory (B), 25 were just satisfactory (C), 63 were just unsatisfactory (D), and 15 were poorly attempted (E), and no reports received the lowest grading (F). The percentage of reports graded as satisfactory (A to C) was 35%. The most common grade was D, followed by C and then B. The description of project and environment was better performed area and identification and evaluation of key impacts, alternatives and mitigation, and communication of results were less performed areas.

Table 3. An overview of the results of quality review of a sample 120 EIARs.

Summary of category grades		A	B	C	D	E	F	N	A–C (%)	D–F (%)
1.1	Description of the development	0	24	85	11	0	0	0	91	9
1.2	Site description	0	22	81	16	0	0	1	86	13
1.4	Environmental description	1	23	51	39	6	0	0	63	38
1.5	Baseline conditions	2	17	35	32	33	1	0	45	55
2.1	Definition of impacts	0	11	34	48	24	3	0	38	63
2.2	Identification of impacts	0	9	32	60	16	3	0	34	66
2.3	Scoping	3	15	28	45	24	5	0	38	62
2.4	Prediction of impact magnitude	1	13	21	37	37	11	0	29	71
2.5	Assessment of impacts significance	0	12	20	37	39	11	1	27	73
3.1	Alternatives	4	20	20	30	31	14	1	37	63
3.2	Scope and effectiveness of mitigation measures	1	16	15	51	30	7	0	27	73
3.3	Commitment to mitigation	0	12	30	36	31	6	5	35	61
4.1	Layout	1	16	40	52	11	0	0	48	53
4.2	Presentation	1	16	34	55	14	0	0	43	58
4.3	Emphasis	0	13	31	53	21	2	0	37	63
4.4	Non-technical summary	1	12	36	53	16	2	0	41	59
Summary of review area grades
1	Description of the development and the environment	1	21	53	44	1	0	0	63	38
2	Identification and evaluation of key impacts	0	10	26	55	26	3	0	30	70
3	Alternatives and mitigation	2	18	16	47	32	5	0	30	70
4	Communication of results	1	15	33	59	12	0	0	41	59
	Overall quality	0	17	25	63	15	0	0	35	65

Introduction effect of JICA guidelines in 2004 and effect on the quality by sector

The distribution in quality and period of four-year intervals was shown in Table 4. The grade B reports were available after the guidelines were introduced in 2004 and the guidelines seemed to improve the quality. The statistical difference was determined by two-sided Spearman’s correlation coefficient by rank test. The p-value was .047* and it could be said that JICA ESC guidelines in 2004 resulted in improved quality of EIARs and the introduction effect of the guidelines was recognized. The distribution in sectors is shown in Table 5. The quality of transport and power seemed to be better than that of other sectors. The statistical difference was determined by upper-sided Kruskal–Wallis test. The p-value was .54 and the difference between the quality and the sectors was not significant even variability existed between sectors. It could be said that the effect on quality by sector was not recognized.

Table 4. Distribution in quality and periods of four-year intervals (n = 120).

Period	A	B	C	D	E	F	Total
2001–2004	0	0	9	26	5	0	40
2005–2008	0	7	8	21	4	0	40
2009–2012	0	10	8	16	6	0	40
Total	0	17	25	63	15	0	120

Table 5. Distribution in quality and sectors (n = 120).

Sector	A	B	C	D	E	F	Total
Transportation	0	10	7	25	5	0	47
Regional development	0	2	4	10	5	0	21
Power	0	4	4	11	0	0	19
Water resource	0	0	5	11	1	0	17
Pollution control	0	1	2	6	2	0	11
Agriculture	0	0	3	0	2	0	5
Total	0	17	25	63	15	0	120

Effect on quality by both processes of alternatives analysis and public involvement

After the application of JICA ESC guidelines in 2004, a number of both processes of alternatives analysis and public involvement increased rapidly (Table 6) and their quality appeared to be better than that of other three groups (Table 7). Assuming that they were key factors in improving the quality of JICA EIARs, the statistical difference was determined by upper-sided Kruskal–Wallis test. The p-value was .000** and it could be said that the effects by both processes of alternatives analysis and public involvement on the quality of reports were recognized. On the other hand, the quality of alternatives analysis process only or that of public involvement process only was not good and no difference was recognized from neither processes. The alternatives analysis process only or the public involvement process only was likely to have no effects on the quality.

Table 6. Distribution in alternatives analysis and public involvement (n = 120).

Groups	2001–2004	2005–2008	2009–2012	Total
Both processes	2	22	25	49
Alternatives analysis process only	17	8	2	27
Public involvement process only	1	3	6	10
Neither process	20	7	7	34
Total	40	40	40	120

Table 7. Distribution in quality, and alternatives analysis and public involvement (n = 120).

Groups	A	B	C	D	E	F	Total
Both processes	0	17	15	15	2	0	49
Alternatives analysis process only	0	0	4	19	4	0	27
Public involvement process only	0	0	1	7	2	0	10
Neither process	0	0	5	22	7	0	34
Total	0	17	25	63	15	0	120

Effect on quality by public involvement

In the case of presence or absence of alternatives analysis, the quality by a number of public involvement stages was shown in Table 8. In the case of presence of alternatives analysis, the quality of EIARs seemed to improve with an increase in the number of public involvement stages, while in the case of absence, no improvement of the quality seemed to be recognized. The statistical difference was determined by two-sided Spearman’s correlation coefficient by rank test. The p-value of the former was .000** and the p-value of the latter was .998. It could be said that the effects of public involvement on quality were recognized in the case of a presence of alternatives analysis and were not recognized in the case of an absence.

Table 8. Distribution in quality and a number of public involvement stages.

Groups	A	B	C	D	E	F	Total
In the case of the presence of alternatives analysis (n = 76)
PI0	0	0	4	19	4	0	27
PI1	0	2	2	10	1	0	15
PI2	0	9	9	4	0	0	22
PI3	0	6	4	1	1	0	12
Total	0	17	19	34	6	0	76
In the case of the absence of alternatives analysis (n = 44)
PI0	0	0	5	22	7	0	34
PI1	0	0	0	4	2	0	6
PI2	0	0	1	2	0	0	3
PI3	0	0	0	1	0	0	1
Total	0	0	6	29	9	0	44

Number of alternatives and evaluation criteria by public involvement stages

With the 76 samples of presence of alternatives analysis as a target, the result of upper-sided Tukey–Krammer test between the number of alternatives and the evaluation criteria, and the number of public involvement stages was shown in Table 9. The number of evaluation criteria increased with the increase in the number of public involvement stages, while the number of alternatives did not. A significant difference was not recognized but when the number of public involvement stages went up, the number of evaluation criteria for alternatives analysis showed an increasing trend in comparison to the number of alternatives.

Table 9. Tukey–Krammer test results of public involvement stages and number of alternatives and evaluation criteria.

Groups	Mean	Standard deviation	n	PI1	PI2	PI3
Public involvement stages and number of alternatives
PI0	3.7	1.7	27	1.2483	1.0525	1.1864
PI1	4.4	3.0	15		0.2978	0.0248
PI2	4.2	3.0	22			0.3047
PI3	4.4	2.1	12
Public involvement stages and number of evaluation criteria
PI0	3.8	4.5	27	0.8324	1.3649	3.6952
PI1	4.8	2.8	15		0.3702	2.6180
PI2	5.3	4.5	22			2.4800
PI3	8.7	9.7	12

Note: n = 76.

A path analysis with SEM

A causal hypothesis using a path diagram stated that alternatives analysis, public involvement, and the number of evaluation criteria would have an effect on the improvement of the quality of JICA EIARs. The causal model of five variables to the overall quality was calculated to add two variables (the number of alternatives and mitigation) (Figure 1). The adaptation indexes including chi-square, goodness of fit index (GFI), adjusted GFI (AGFI), standardized root mean square residual (SRMR), root mean square error of approximation (RMSEA), normed fit index (NFI), and comparative fit index (CFI), were very satisfactory. The degree of freedom (df) was six and the p-value was .950, which was very high. The coefficient of the determination (R²) of the overall quality was .74. The causal influence of the model explained 74% of the fluctuations. The residual (e) explained fluctuations by means of various causes outside of the model. The positive path coefficients implied the effect and the overall quality increased. The causal model showed not only the direct effects but also the indirect effects of variables on the overall quality. The alternatives analysis showed that the total effect on the overall quality was .76. The direct effect was .30 and the indirect effect was .46 (= .42 × .10 + .71 × .56 + .71 × .24 × .10). The public involvement showed the direct effect only on the overall quality, which was .10.

Figure 1. Causal model with path coefficients.

Discussion

Effectiveness of alternatives analysis and public involvement on quality

The JICA ESC guidelines in 2004 may have effects on improving the quality of EIARs and the alternatives analysis and public involvement may be key factors in improving the quality. As a result of the tests between four groups by alternatives analysis and public involvement, the p-value was .000** and the both processes of alternatives analysis and public involvement could be confirmed to increase the quality. The two processes needed to work together. The alternatives analysis process only or public involvement process only could not have any effects on the quality. Research on alternatives analysis showed the decision-making process (Janssen 2001; Sólnes 2003; and Hajkowicz 2008) and the insufficient range of alternatives setting (Steinemann 2001; and Smith 2007). Research on public involvement in EIA showed the improving quality of decisions as one benefit of public involvement (National Research Council 2008, p. 50). However, there are very few research focused on the linkage between alternatives analysis and public involvement and little is known about the effects of both processes on improving the quality of EIARs.

When the number of public involvement stages went up, the quality was improved (p = .000**) and a number of evaluation criteria for alternatives analysis showed an increasing trend. It is possible that the public involvement increased the project proponents’ awareness of environmental and social considerations. As a result, the number of evaluation criteria tended to increase. Widening the range of evaluation criteria enabled an increase in the amount of secondary environmental and social information so as to account for a justification of projects and to provide a reason for selecting the best option at the scoping and draft reporting stages consultation. This may result in an improvement in the quality of JICA EIARs. Increasing the number of evaluation criteria may be one concrete benefit of public involvement.

Causal relationship between alternatives analysis and public involvement

The causal model with path coefficients may verify the effectiveness of variables on the overall quality of JICA EIARs. The effect of the number of evaluation criteria was .44 and was higher than that of alternatives (.35), which ensured that the number of evaluation criteria was more likely to have an effect on quality compared to one of alternatives. The indirect effect was positive result of the path analysis. The path from alternatives analysis to public involvement with .42, showed the linkage between alternatives analysis and public involvement, and provided clear evidence that the overall quality of both processes was significantly higher than for the other three groups (Table 7). This indirect effect explained the fact that a good alternatives analysis produced good public involvement and good public involvement resulted in an improvement in the overall quality. This path with .42 could also be evidence to show that the discussion of alternatives was “the heart of the environmental impact statement” ([CEQ] Council on Environmental Quality 1978).

The total effect of alternatives analysis on the overall quality was .76. On the other hand, the effect of public involvement was only .10. What does this difference mean? One interpretation is that public involvement may have a role in improving alternatives analysis through collecting secondary environmental and social information, rethinking the substance of alternatives, and maintaining the credibility and legitimacy of a project and a selected option. Another indirect effect of alternatives analysis (.71) was that a good alternatives analysis produced a good mitigation. In other words, it could be said that alternatives analysis was an analysis of mitigation options, for example, a zero option meant an avoidance option. Each alternative mixed mitigation measures including avoidance, minimization, and replacement of various impacts. Through the alternatives analysis process, a better mitigation option could be selected. There was one path from mitigation to public involvement and it was conceivable that public involvement may also improve mitigation. The comparison of the four groups with the results of the statistical test showed the effectiveness of both processes of alternatives analysis and public involvement; the causal model showed the causal relationships between them through indirect effects and path coefficients. A good alternatives analysis may be a condition for good quality JICA EIARs. Public involvement may be an essential process for improving alternatives analysis and mitigations, even the value of the effect was not large.

Methods for improving the quality of JICA EARs

Summarizing the above discussion, the comparison of groups and tests and the causal model provided examples of concrete knowledge that could be used to improve the quality of JICA EIARs. The use of alternatives analysis and two stages of public involvement could be one standard for satisfactory quality because 18 of 22 reports (about 80%) were assigned B and C grades (Table 8). The mean value of alternatives and evaluation criteria was 4.2 and 5.3, respectively (Table 9). The methods for improvement may be: (1) alternatives analysis and two-time public involvement at the scoping and draft reporting stages; and (2) setting five alternatives, and more than six evaluation criteria for alternatives analysis.

Comparison with other findings by the Lee-Colley review package

Previous studies used the portion of grades that were satisfactory (A to C) and unsatisfactory (D to F), and then went on to discuss the improvement in quality (Wood et al. 1996; Cashmore et al. 2002; Canelas et al. 2005; Momtaz & Kabir 2013; and Sandham et al. 2013). For example, the quality of EIARs in Portugal was compared in three periods (1990/1992, 1994/1995, and 1998/2003). The percentage of satisfactory was 40, 71, and 78%, and the study concluded the quality had the improving trend although it seemed to be gradually stabilizing (Canelas et al. 2005, p. 222). The statistical test was not utilized. The p-value calculated by two-sided Spearman’s correlation coefficient by rank test using data of that study, was .000**. The improvement of the quality of EIARs in Portugal was recognized by statistical test. The conclusion was judged to be suitable. The statistical test may be a useful and essential tool for judging the quality improvement.

Many constraints were identified in previous research. A lack of political will, limited published research, and EIA practitioners not being independent from developers were major constraints in South Africa (Sandham & Pretorius 2008). A shortage of study time, inadequate baseline data and access to data, the attitudes of consultants and proponents, a lack of EIA experts, defective service procurement processes, a lack of adequate funds, weak terms of reference, and a lack of EIA team members were identified in Bangladesh (Momtaz & Kabir 2013). A proposed solution for improving the quality of EAIRs was more quality review research in South Africa. No concrete methods were proposed in the case of Bangladesh. It would be very difficult to overcome the above-mentioned constraints and to improve the quality in developing countries. The statistical analysis allowed an analyst to find the key factors for improving the quality of JICA EIARs and to propose concrete methods.

Conclusions

The study analyzed the quality of a sample of 120 JICA EIARs using the Lee-Colley review package, statistical tests and pass analysis with SEM. Using the analysis results, the study was able to quantitatively explain the effectiveness of alternatives analysis with a wide range of evaluation criteria and two-time public involvement at the scoping and draft reporting stages for improving the quality of JICA EIARs. In practical terms, working both processes together with five alternatives and more than six evaluation criteria could be specific guidelines for good quality EIARs under constraints of developing countries. The statistical analysis and pass analysis with SEM were very useful tools for finding and verifying key factors for improving the quality of EIARs. Compared with previous findings by the Lee-Colley review package, this study was able to focus more on concrete methods for improving quality.

The study showed that the discussion of alternatives was the heart of the environmental impact statement based on clear evidence of the indirect effect between alternatives analysis and public involvement in the causal model. Such results have not been previously shown and would be beneficial for enhancing justifications and understandings of alternatives analysis and public involvement within EIA. This study suggested the possibility of improving the quality of not only JICA EIARs but also EIARs in developing countries even under constraint of lack of EIA expertise and related information. It may be possible to improve the quality of EIARs more by improving both processes of alternatives analysis and public involvement and their linkage. Further research is needed to explore alternatives analysis and public involvement in more depth as well as improve the effectiveness of their linkage via more case studies.

Disclosure statement

No potential conflict of interest was reported by the authors.

Acknowledgments

This work was supported by the JICA Research Institute. We are grateful to the anonymous reviewers for their valuable comments, which improved the manuscript.