More efficient project execution and evaluation with logical framework and project cycle management: evidence from international development projects

Abstract

The importance of international development projects in the field of international aid is growing. Given the characteristics of these projects, some specific methodologies have been developed, such as project cycle management (PCM) and the logical framework (LF) to efficiently carry out social impact assessment and project execution. However, few studies have analyzed the diffusion of these methodologies, especially in the context of non-governmental organizations (NGOs). With a survey of 109 Project Managers working for Italian NGOs, we show that PCM and LF are extensively adopted by NGOs compared with other standard methodologies (e.g. PMBOK Guide, IPMA). Moreover, we show how the adoption of PCM and LF is related to the level of knowledge of such tools by project managers, thus highlighting the importance of training. Finally, in line with previous studies, we find that project managers in NGOs believe that such tools are not easy to use but optimize the (social) impacts of the projects.

Keywords:

• International development
• logical framework approach
• empirical study
• non-governmental organization
• project cycle management

Introduction

International development (ID) projects are gaining increasing importance in the field of international aid of developing countries. Different from emergency projects, which provide immediate assistance to populations hit by wars or natural disasters (e.g. Durning 2014; Gore & Fischer 2014), ID projects usually take place in a more stable context, their aim being to improve living conditions in terms of economy, education, or health. Recent statistics confirm a growing trend in the amounts of money and human capital employed in this field (Diallo & Thuillier 2005; OECD 2008; 2009b), and an increasing number of awareness campaigns and initiatives seek to heighten interest in projects of this kind and their results.

In spite of its large size, the international aid industry, and ID projects in particular, have turned poor performance into rule rather than exception (Ika et al. 2012). In fact, according to a recent McKinsey-Devex survey, 64% of donor-funded projects fail in achieving their intended impact due to poor project planning and management skills (Hekala 2012) and combination of structural/contextual and institutional/sustainability problems (Ika et al. 2012).

This fact has raised the need for better impact assessment and accounting systems to be adopted by non-governmental organizations (NGOs) (e.g. Ebrahim 2003a, 2003b) in order to improve and control the ‘social impact’ of their interventions (Becker & Vanclay 2003; Golini & Landoni 2014). Vanclay (2003) defines social impact assessment (SIA) as ‘the processes of analysing, monitoring and managing the intended and unintended social consequences, both positive and negative, of planned interventions (policies, programs, plans, projects) and any social change processes invoked by those interventions’ and underlines how more managerial efforts should be made to include such an assessment into each phase of the intervention. Often, in fact, project appraisal (and a fortiori evaluation) is performed too late in the project cycle, so that it becomes only a formal activity without a real effect in guiding the intervention (Mosley 2001). Even though some of the tools and approaches are similar, SIA differs from the environmental impact assessment (EIA), which aims at ‘protecting the people and environment from the consequences of reckless or inadequately informed policy and decisions’ (Caldwell 1988). In other words, while EIA is a necessary step for a broader public decision, SIA is rather related to evaluate the main objective of the project. Of course, social projects can also include environmental aspects.

The literature has also identified projects’ Critical Success Factors (CSF) as those aspects that must go well to ensure a project’s success, and which must therefore be given special and continual attention (Hermano et al. 2013). In particular, some authors (e.g. Baccarini 1999; Cooke-Davies 2002) argue that the success of a project should be divided into two separate concepts: (1) project management success, which is related to the traditional concepts of cost, schedule, and quality (the well-known ‘iron triangle’); and (2) project success, which concerns stakeholders’ satisfaction and achievement of the company/organization’s strategic goals. A project’s success is not a simple unitary concept but depends on the stakeholder assessing that success (de Wit 1988; Baccarini 1999; Lim & Mohamed 1999; Freeman & Beale 1992, p. 8).

This is in line with other studies showing that Project Management (PM) tools and techniques play an important role in helping projects to succeed, but they do not guarantee their success (e.g. Munns & Bjeirmi 1996; Mingus 2002).

Another aspect of project success that has been studied is the dichotomy between a static view of project success and a dynamic one. Khang and Moe (2008) designed a project success framework in which the factors crucial for achieving project success vary according to the phase of the life-cycle in which the project is evaluated.

With a better and more integrated impact assessment, and the alignment with the ‘Critical Success Factors’ in mind, ad hoc methodologies and tools have been developed. Among the latter, this study focuses in particular on project cycle management (PCM) and the logical framework (LF). As highlighted by Landoni and Corti (2011), these tools are used by the most important governmental development agencies. These tools have also been developed in order to establish a connection between social impact assessment and project management (Newcomer et al. 2013). They are incorporated into the project planning, monitoring, and appraisal processes throughout the entire life cycle (Khang & Moe 2008). These specific tools help governmental agencies to manage ID projects in pursuit of the long-term objectives identified while keeping their social impact high (Golini & Landoni 2014).

Surprisingly, however, the literature is limited in regard to the adoption of these methodologies in NGOs. In fact, there is no evidence on the adoption of specific PM methodologies or tools in the non-governmental sector.

Given this gap, the purpose of this paper is to analyze, by drawing on a survey of project managers working for Italian NGOs, the extent to which PCM and the LF are used to manage ID projects in the non-governmental sector.

The paper is organized as follows. In the literature review section, we describe the peculiarities of the non-governmental sector and we discuss the main characteristics of ID projects and the tools used to manage and evaluate them. Next, we state our research questions and provide information about the methodology and data employed in our research. We then set out the results obtained and, finally, draw our conclusions, highlighting how this paper contributes to the existing literature and project management and assessment practice.

Literature review and research objectives

Despite the universalist nature of project management methodologies, different contexts exhibit different approaches to project management (Hanisch & Wald 2012). As a consequence, there has been a call to investigate the adoption of project management methodologies in different environments (e.g. Marshall & Fischer 2006). This is a gap that researchers and practitioners are trying to fill, and which over time has led to the adaptation of project management methodologies to specific contexts (Besner & Hobbs 2008a; Golini et al. 2015). However, many gaps still remain in knowledge about how NGOs engage in project management for ID projects.

NGOs and ID projects

NGOs are characterized by a specific focus on human development and poverty reduction. NGOs are private organizations, independent from governments and their policies, which operate with no-profit purposes to improve the living conditions of poor populations (Vakil 1997). Within this broad definition, the literature has identified some additional characteristics of NGOs (Raimondi & Antonelli 2001). Firstly, they normally apply a participative approach, involving the beneficiaries in the key aspects of the development process. Moreover, they tend to be ‘democratic’ rather than ‘hierarchical’ organizations. Finally, they tend to diversify their financing sources in order to avoid being influenced by the sponsors.

Today, NGOs have an important role in reaching the poorest populations, and they provide them with effective help, sometimes with the endorsement of governments (Koch et al. 2009). More than 40,000 NGOs are estimated to operate worldwide (Kovach et al. 2003).

Furthermore, their importance is growing because of:

•	the successes achieved by some NGOs (Brown & Kalegaonkar 2002);
•	the limitations of governments as helping agents (Hyden 1998; Lindberg et al. 1998);
•	the involvement of private citizens (Putnam et al. 1994; Woolcock 1998).

When NGOs manage ID projects, other specificities have to be considered. In fact, ID projects have a number of particular characteristics (Kwak 2002; Youker 2003; Diallo & Thuillier 2005; Khang & Moe 2008; Ika & Hodgson 2010; Ika et al. 2010; 2012; Ika & Saint-Macary 2012) that entail the use of specific project management tools and approaches. ID projects are at once social, political, and technical undertakings. They are social because they aim directly or indirectly to improve the health, wealth, or level of instruction of the population living in one area or of specific target groups (e.g. women, children). They can also aim at strengthening the traditional social/cultural practices or foster integration and equity. At the same time, ID projects are political because the choice of project options, such as locations or target groups, is often a political decision made by ID agencies, donors, national political leaders, and policy-makers (Diallo & Thuillier 2004, 2005). Finally, ID projects are also technical, as they often require hard (e.g. engineering) and soft (e.g. cultural mediation) skills applied at the same time.

Moreover, ID projects’ objectives are less tangible and measurable than standard business ones, and they have a not-for-profit nature. Furthermore, a higher number of stakeholders are involved than in other kind of projects, and these stakeholders have blurred roles. ID projects require, in fact, at least three separate key stakeholders: the funding agencies, who pay for but do not receive project deliverables; the implementing units, which are involved in the execution of projects; and the target beneficiaries, who expect some benefit from them (Khang & Moe 2008). Diallo and Thuillier (2004, 2005) proposed a more complex network comprising eight project stakeholders: the national project coordinator (also known as the national project coordinator/director) which manages the project; the project team which executes the project; the ID agency which supervises the project; the national supervisor; a steering committee; subcontractors and suppliers of goods and services; beneficiaries; and the population at large. Ranasinghe (2008) suggested grouping ID project stakeholders into four categories: those directly affected; those indirectly affected; government and public sector departments; and donors, consultants, managers and private business persons. Furthermore, several countries are involved, which raises issues in terms of cultural differences, physical distance and coordination: projects are defined and designed in developed countries but implemented in a managerially, economically, and politically different context. This is particularly the case of ID projects funded by ID agencies, especially multilateral ones, like the World Bank, the United Nations, and the European Union, and bilateral ones like the United States agency for international development (USAID), the Canadian international development agency (CIDA), French Cooperation, and many other governmental and non-governmental organizations. Finally, project environments in developing countries are often difficult because of corruption, poor infrastructure and a lack of resources (Ika 2012). The lack of market pressures and the intangibility of project objectives make ID projects good targets for political manipulation: donors and political leaders may push pet projects for political gains (Khang & Moe 2008; Bokor 2011).

All these conditions make the critical success factors of ID projects unique. Following up on Belassi and Tukel (1996), Hermano et al. (2013) group the critical success factors into four different areas that cover almost all the possible aspects affecting a project’s performance: (1) factors related to the project; (2) factors related to the team (including the project manager); (3) factors related to the organization; (4) factors related to the external environment. What emerges from Hermano et al. (2013) study is that, although there are fundamental similarities in CSFs among different types of projects, adopting a project-specific approach is necessary to ensure success.

As a consequence, it is essential to have proper methodologies with which to manage ID projects. Researchers and practitioners agree that project management can be applied to different contexts, but also that some adaptations are necessary industry by industry (Besner & Hobbs 2008b). As highlighted by Hanisch and Wald (2012) ‘no project can be studied comprehensively without considering its context: the congruence of a project to the external contingencies is considered to be a factor influencing the effectiveness of the temporary organization’.

Certainly, none of these conditions is necessarily unique to ID projects and project management. However, all of these conditions together challenge the contract-based precepts and modus operandi of standard project management (Ika 2012).

Given these peculiarities of ID projects, specific tools have been developed to manage them. Firstly, in 1970, Baum introduced the project cycle concept into ID projects (Baum 1970). The project cycle breaks down a project into phases that connect the beginning of the project to the end (Figure 1). PCM is a broad methodology that follows the project in its three core phases: planning (that includes appraisal), monitoring and control, and closing (including ex-post evaluation). The presence of a continuous assessment of the project in all three phases is one of the aspects that make PCM so interesting (Ahonen 1999).

Figure 1. Baum’s project cycle (Baum 1978).

PCM is a framework rather than a tool in itself. As a consequence, different tools have been developed within PCM (Biggs & Smith 2003). The best known of them is the LF, which today is widely used and often considered a stand-alone tool (Couillard et al. 2009). The LF was developed in 1969 by Fry Associates and Practical Concepts, Inc. for the United States Agency for International Development (USAID) (Solem 1987). In its original form, the LF is a 4 by 4 matrix crossing the goals, purpose, inputs and outputs of the project with the sources of verification and assumptions (Figure 2). The objective is to provide a synthetic view of the project for all the stakeholders and support the design, planning, management, evaluation, and communication of the project (Coleman 1987; Gasper 1997).

Figure 2. Example of logical framework.

The requirement of a monitoring and evaluation system at all levels of the project, imposed by the LF logic to the project designers and in the project implementation phase, is perceived as essential by the European Commission, that requires the LF approach to be adopted at each stage of the project cycle, from identification to ex post evaluation (Akroyd 1999). The importance of tools to support ex-post evaluations has been recently highlighted in the literature (e.g. Nicolaisen & Fischer 2016). The IFAD (International Fund for Agricultural Development), the FAO (Food and Agriculture Organization) and the World Bank, since 1987 (Casley & Kumar 1987), demand that when projects are prepared and appraised, the monitoring and evaluation function of LF is to be perceived primarily as an essential part of the process of project management and implementation, created to provide information required by the project manager. A secondary but important function of the LF is to harmonize the information flows between the project manager, the staff, the executing agency and external funding agencies (Akroyd 1999).

As reported by Landoni and Corti (2011), PCM and the LF are adopted by some of the most important governmental agencies. However, differences in how these methodologies are used testify to a lack of standards in the management of ID projects. For example, AusAID (the Australian Government’s overseas aid program) does not explicitly mention PCM, even if it works with a similar framework based on six phases. JICA (Japan International Cooperation Agency) instead adopts the LF, adding pre-conditions to the columns.

This empirical evidence is in line with criticisms made of PCM and the LF by researchers over the years. Crawford and Bryce (2003) underline that problems with the LF approach stem from four main issues: (1) the absence of a time dimension; (2) the inappropriateness of assigning efficiency-level objectively verifiable indicators; (3) the inadequacy of the means of verification; and (4) the static nature of the logframe. The literature argues that the LF has unclear terminology, unclear links between its levels, and a lack of stakeholder involvement (Couillard et al. 2009). A lack of integration with other project management methodologies is also cited as a major problem, because PCM and the LF cannot replace completely traditional project management tools (e.g. Work Breakdown Structure, Gantt diagram).

This has led to several reformulations of PCM and the LF, and some recommendations for further improvements (e.g. Gasper 2000; Couillard et al. 2009).

Research objectives

Despite the highlighted uniqueness of NGOs and ID projects, and despite the presence of specific methodologies and tools, limited attention has been paid to these topics (Youker 2003). In particular, we identified three aspects worth investigating in our research.

First, as previously noted, there is evidence in official documents of the adoption of PCM and the LF by governmental agencies (e.g. USAID, JICA, etc.), while there is limited evidence of the extent of this adoption by NGOs. Furthermore, governmental agencies adopt these tools in different ways, seeking to overcome their limitations (Landoni & Corti 2011).

For this reason, we investigated the extent to which NGOs adopt such project management methodologies by focusing on the project managers of ID projects. As a matter of fact, NGOs are not compelled to follow a particular project management practice or standard. They do not have to adopt international and general methodologies (e.g. those developed by the Project Management Institute – PMI, the International Project Management Association – IPMA, etc.) or specific ones. They may be required to highlight PCM references and to provide a LF when submitting project proposals to international governmental agencies and, in some cases, when reporting their intermediate and final results. However, also in these cases, the tools may be used only for a short time to comply with the requests and being abandoned later during the project execution.

To summarize: our first objective consisted in understanding the extent to which PCM and the LF are adopted in ID projects by project managers working in NGOs (Objective 1).

Second, there are several factors that can be related to the greater and differing adoption of the PCM and LF tools in ID projects (Besner & Hobbs 2008a). Our second objective thus focused on determining the factors that most significantly affect the level of adoption of PCM and the LF (Objective 2).

For instance, project size is related to the organizational maturity in terms of capabilities (Besner & Hobbs 2008a). The number of projects and project duration are other influential variables (White & Fortune 2002). Furthermore, the size of the NGO and the age of the project managers are other relevant variables to consider (White & Fortune 2002; Besner & Hobbs 2008b). Since NGOs are not compelled to adopt particular methodologies, we did not expect all project managers to have in-depth knowledge of standard project management methodologies and tools. We consequently included the level of knowledge of such tools among the variables to analyze. Finally, several studies (e.g. Crawford et al. 2006) agree that not all project management techniques are adopted for every project; rather, they may be selected according to the circumstances, bearing in mind that Besner and Hobbs (2004) demonstrated that practitioners, regardless of the project’s characteristics and context, almost invariably use some PM tools and techniques, the bulk of which have different levels of usage according to the type of project (Ika et al. 2012). Accordingly, we hypothesized that project managers in NGOs mix the LF and PCM with other project management methodologies. Finally, we identified a series of factors that can theoretically influence the adoption of PCM and the LF (size of the NGO, number of projects, size of the projects, years of experience of the project managers, level of knowledge of PCM (LF), use of other project management methodologies), and we examined their impact on the level of adoption.

As a third point, the literature suggests that, although these tools, and especially LF, have several important and acknowledged benefits, they may also have limitations in terms of ease of use and usefulness. Couillard et al. (2009) highlight that the LF has unclear terminology, unclear links between levels, lack of stakeholder involvement, and a lack of integration with other project management methodologies. Other limitations have been found by researchers. Gasper (2000) states that the LF is sometimes used more for formal purposes than as a guide for the project. Moreover, the terms and the logic employed are not easy to understand, and this may cause inconsistencies or oversimplifications. Furthermore, LF is often applied by following a routine, and this reduces opportunities to learn and improve the LF itself. Bell (2000) also underlines that LF has been adopted in the past without any evolution and adaptation to the context. Finally, Smith (2000) criticizes the binary logic that characterizes the LF and that would require perfect knowledge of the problem.

These limitations are also apparent in the new methodologies based on life cycle assessment and developed for NGOs operating in international cooperation (e.g. PM4DEV, PM4NGO with its PMD Pro1 body of knowledge). PMD Pro1 was specifically created in order to develop a new framework for methodologies and techniques already explained in general bodies of knowledge such as PMBOK® or ICB®, but also to take the peculiarities of the international aid industry into account.

For these reasons, our third objective was to understand how project managers evaluate PCM and the LF and what factors affect their evaluations (Objective 3). On the one hand, the level of satisfaction with these tools is an important issue because it can suggest whether the tools should be revised or whether their diffusion within the ID project community should be promoted. On the other hand, we considered it interesting to understand what prompts a better or worse evaluation by examining the role of the above-described influencing factors.

Methodology and data

Survey design

In order to achieve the research objectives, we used data collected in 2010 on a sample of 109 project managers working for 84 Italian NGOs, including some organizations that are based in Italy and are autonomous but are also linked with NGOs born in other developed countries. Italian NGOs play a significant role in the international context. A 2008 study (Link 2007 2008) highlighted that they are active in more than 100 countries worldwide, with a strong presence in Africa. The capital flowing through Italian NGOs in 2008 amounted to € 1 billion, of which 39% came from private citizens or companies (OECD 2009a). Furthermore, there is a growing trend in terms of number of NGOs active on ID projects and in terms of funding devoted by these NGOs to international aid.

Even if other non-profit organizations devoted to international cooperation exist in Italy, we selected only those accredited by the Italian Ministry of Foreign Affairs in order to ensure the quality of our sample. An accredited NGO must be financially transparent and should fulfill various requirements in terms of personnel (experience, contracts), administration and controls.

In 2009, about 250 NGOs were accredited at the Ministry of Foreign Affairs.

Several project managers per NGO were contacted by telephone, and if they showed interest in the research, a questionnaire was sent to them. We sent out around 300 questionnaires and we had a response rate of 30%. To reduce possible sample bias, we presented the survey as a general study on project management practices in ID projects, and we clearly stated that there were no prerequisites, previous knowledge, etc. for responding to the survey, and that the data would be treated anonymously.

The questionnaire was designed consistently with the research objectives and the literature. The first section collected general information about the NGO (total income, average size of the projects), and the respondent (year of birth). The remaining questions focused on the projects managed by the project managers interviewed and their knowledge and adoption of project management methodologies, PCM and the LF. Finally, we asked the project managers to evaluate PCM and the LF in terms of their usefulness and ease of use. The sample is described in Table 1 in terms of size of the NGO, average project size, and number of projects managed by the project managers during 2008–2009. The years of birth of the project managers ranged from 1946 to 1984, with an average value of 1970. Almost half (45%) of the project managers interviewed were female.

Table 1. Sample description.

(a) Size of the NGO (total income × 1000 €)	N	%	(b) Size of the projects	N	%	(c) Number of projects	N	%
Less than 500 €	17	16	Less than 50,000	13	12	1–10	78	72
500 €–2,000 €	45	41	50,000–150,000	29	27	10–20	13	12
2,000 €–10,000 €	34	31	150,000–500,000	42	39	20–50	7
10,000 € +	12	11	500,000+	25	23	50+	8	7
NA	1	1	NA	0	0	NA	3	3
Total	109	100	Total	109	100	Total	109	100

Measures

To perform our analyses, we considered the variables described in Table 2. Some variables required transformation from categorical to ordinal in order to allow statistical analyses. Different variables were used as dependent and independent for the different analyses.

Table 2. Variables description.

Variables	Original scale		Transformed scale
Dependent/Focus variables
• Level of knowledge of PCM	Categorical	Tool not known	Ordinal	1
		Low		2
		Medium		3
				4
• Level of knowledge of LF
		High
• Adoption of PCM	Calculated variable: Ratio between Number of projects adopting PCM (LF) and Number of projects
• Adoption of LF
• Evaluation of PCM usefulness	Ordinal	1: not useful at all	No transformation
		2: not very useful
• Evaluation of LF usefulness
		3: quite useful
		4: very useful
• Evaluation of PCM ease of use	Ordinal	1: very difficult	No transformation
		2: not easy
		3: quite easy
		4: very easy
• Evaluation of LF ease of use
Independent/explanatory variables
• Total number of projects	Number		No transformation
• Number of projects adopting PCM
• Number of projects adopting LF
• Size of the NGO	Categorical	Less than 500 €	Ordinal	1
		500 €–2,000 €		2
		2,000 €–10,000 €		3
		10,000 €+		4
• Size of the projects	Categorical	Less than 50,000	Ordinal	1
		50,000–150,000		2
		150,000–500,000		3
		500,000+		4
• Year of birth	Number		Log transformation
• Use of other Project Management standards	Categorical	PMBok (PMI)	(Binary)	1
		IPMA		1
		PRINCE2		1
		Others		1
		No one		0
• Type of adoption of PCM	Categorical	As is Picking only some tools and adapting them	No transformation

In particular, for our first objective (determining the extent to which PCM and the LF are known and adopted), we considered the level of knowledge and the level of adoption (calculated as the ratio between the number of projects adopting PCM/LF and the total number of projects).

Next, for our second objective (determining what factors can affect the level of adoption), we considered the level of adoption of PCM/LF as a dependent variable and a set of other factors (total number of projects, size of the NGO, size of the projects, years of birth of the project managers, use of other project management methodologies) as independent variables.

Finally, for our last objective (determining how project managers evaluate these techniques and what factors can affect their evaluations), we first analyzed how the project managers assessed PCM and the LF in terms of their usefulness and ease of use. We then used these as dependent variables and the same set of variables as previously employed (i.e. total number of projects, size of the NGO, size of the projects, years of birth of the project managers, use of other project management methodologies) as independent variables.

Results and discussion

Adoption of project management methodologies (Objective 1)

We first investigated the level of adoption of the LF and PCM by the project managers working in the NGOs. In so doing, we also wanted to verify whether these methodologies were more frequently used than other general PM methodologies (e.g. PMI and IPMA).

We found that general PM methodologies (e.g. PMI, IPMA, etc.) were used in only 7% of the projects. This shows that the project managers of ID projects belong to a community separate from the traditional project management community and which uses specific tools. This may be due (a) to a belief that the traditional methodologies are not suitable for ID projects and cannot be applied to ID projects without proper adaptation (Khang & Moe 2008); (b) to a low level of knowledge about these methodologies and limited professionalization of the project managers of ID projects. For example, most of the ID professionals at the World Bank are ‘accidental’ project managers, because they have project and program management responsibilities but lack any formal project management education and background (Ika 2012).

By contrast, as shown in Table 3, we found that only about 10% of the respondents did not adopt PCM and about 6% of them did not adopt the LF. This means that more than 90% of the project managers interviewed had adopted the LF or PCM for at least one project during the years 2008–2009. On average, PCM was adopted in 74% of the total projects managed, and the LF in 81% of the projects, testifying to their widespread use (the difference between PCM and LF adoption was statistically significant with sig. = 0.002). This result is interesting because the LF was created as a part of the PCM, but our results show that it is also used independently. More in detail, Table 3 shows that 48% of the project managers used the PCM for more than 75% of their projects, while 61% of the project managers used LF for more than 75% of their projects. We may therefore conclude that PCM and the LF were widely adopted by the project managers of the NGOs surveyed, but also that PCM and the LF were not adopted for all projects. This means that the project managers decided to use them case by case: these tools were not always considered suitable or convenient for the management of ID projects.

Table 3. Level of adoption (calculated as the ratio between Number of projects adopting PCM/LF and Number of projects).

	Adoption of PCM		Adoption of LF
	N	%	N	%
0% (non adopters)	10	10	6	6
1–25	10	10	7	7
26–50	16	16	13	13
51–75	17	17	14	14
76–100	49	48	62	61
Total	102	100	102	100

The fact that the majority of the project managers used both PCM and the LF for more than 75% of the projects also suggests the joint adoption of PCM and LF. In fact, we found a high and significant correlation (Pearson correlation = 0.615, sig. = 0.000) between LF and PCM adoption. More in detail, 92 project managers adopted PCM and LF for their projects, while 4 adopted only LF, 0 only PCM, and 6 neither PCM nor LF. This may be due to the fact that LF was considered a part of the PCM.

We also considered the type of adoption of PCM (‘as is’ or ‘picking only some tools and adapting them’). Interestingly, 77 project managers out of 92 (i.e. 84%) picked and adapted tools belonging to PCM rather than using the methodology ‘as is’. This suggests that PCM and the LF need to be adapted by users to different contexts, since the ‘as is’ application was very rare.

Factors associated with a higher level of adoption (Objective 2)

In order to achieve the second research objective, we regressed (Table 4) the Adoption of PCM/LF on the following set of independent variables: Size of the NGO, Number of projects, Average size of the projects, Age of the project manager, Use of other project management tools, Knowledge of PCM/LF.

Table 4. Determinants of the level of adoption (Standardized regression coefficients).

	Adoption of PCM	Adoption of LF
Size	−.047	.025
Number of projects	−.105	.010
Size of the projects	.166	.315**
Year of birth	.114	−.063
Level of knowledge of PCM (LF)	.411**	.536**
Use of other PM standards	.056	.086
R-square	.261	.438

^* sig. < 0.050

^** sig. < 0.000.

For PCM and the LF, we found a significant positive impact of the level of knowledge on their adoption. For the LF, also the size of the projects was significant. It is interesting that there was no relationship between the size of the NGO and the level of adoption. This is somewhat counterintuitive because we expected larger NGOs to be more structured for management of their projects. Further investigation is therefore needed to understand whether NGOs use other methodologies or whether it is up to the project manager to decide on their use. Nor did the number of projects affect the level of adoption. This may be related to the fact that these methodologies are usually used at the individual project level, rather than at the program level.

The size of projects was instead related to the greater adoption of the LF, but this relationship did not hold for PCM. This is interesting because larger projects are also those more likely to be subject to strong pressures by stakeholders for good project management: for instance, to be compliant with the budget. Only the LF, typically used at the beginning of projects, was considered a key methodology for larger projects.

The age of the project manager was not related to the level of adoption. We acknowledge that age is different from years of experience, but we deem it a reasonable proxy (El-Sabaa 2001). On the other hand, the level of knowledge was strongly related to a higher level of adoption. This last result is not surprising, but it stresses the need for NGOs to train their project managers in order to foster the adoption of such methodologies.

Finally, the use of other project management methodologies (a binary yes/no variable) was not significant, meaning that their use did not drive the adoption of PCM and the LF.

PCM and LF evaluation (Objective 3)

To conclude our analyses, we considered how the project managers evaluated PCM and the LF (Table 5). Scores for PCM and the LF were very similar. On average, the project managers found PCM and the LF quite useful (3.3 out of 4) in terms of (social) impacts of the projects, but not particularly easy to use (2.8 out of 4). This highlights that greater effort should be made to render these methodologies more suitable for and accessible to NGOs. One Project manager, in the notes of the questionnaire, underlined that ‘LF is useful because it provides guidelines to identify indicators and data to measure the impact’; another one explained that ‘LF allows to identify the problems, the objectives, the results and the necessary activities: it provides a clear, transparent, and logical process to follow and to evaluate the project’.

Table 5. Evaluation scores for PCM and LF (scale from 1 to 4).

	PCM	LF
Evaluation of usefulness	3.27	3.35
Evaluation of ease of use	2.80	2.79

The evaluation scores were also highly correlated (Pearson’s sig. was always 0.000 for every duplet), meaning that those respondents who appreciated PCM’s usefulness and ease of use also appreciated the LF. Moreover, the evaluation of PCM and the LF showed an interesting pattern of correlations with the level of adoption. In particular, the adoption of PCM was not related to its evaluation, while the adoption of the LF was positively correlated to its evaluation in terms of ease, meaning that there is a learning effect on LF, but not on the methodology as a whole.

As a final analysis, in Table 6, we regressed the evaluations given to the LF and PCM with the same independent variables introduced in Table 4. As regards, PCM’s usefulness and ease, there were no significant relations. For the LF, only the level of knowledge was related to a higher evaluation of both usefulness and ease of use. This result again shows the importance of investment in the training of NGO project managers to use these tools. Furthermore, the analysis highlights that PCM and the LF can be used with satisfaction by project managers dealing with large or small projects and by NGOs of different sizes.

Table 6. Determinants of the evaluations (Standardized regression coefficients).

	PCM Usefulness	PCM Easiness	LF Usefulness	LF Easiness
Size	.034	.078	.026	−.033
Number of projects	.171	−.050	.033	.010
Size of the projects	.013	−.126	−.051	.053
Year of birth	.104	.101	.011	−.051
Level of knowledge of PCM (LF)	.086	.199	.227*	.313**
Use of other PM standards	−.145	−.090	−.135	.044
R-square	0.062	0.059	0.064	0.100
Total cases	90	88	93	91

^* sig. < 0.050

^** sig. < 0.000.

Conclusions

By drawing on a survey of 109 Italian project managers working for NGOs and dealing with ID projects, this paper has studied the adoption and evaluation of project management tools in NGOs. As a matter of fact, despite the economic and social importance of this topic, few contributions in the literature have previously addressed it.

Several results emerge from the analyses. First, international methodologies (PMI, IPMA, etc.) are not generally used, while methodologies and tools designed for governmental agencies to manage ID projects (i.e. PCM and the LF), are extensively adopted by NGOs. More than 90% of the project managers surveyed had adopted the LF or PCM for at least one project during the years 2008–2009. The level of adoption of PCM and the LF (i.e. in how many projects on the total they had been used) was related to the level of knowledge of these tools, but not to the size of the NGO, nor with the number of projects managed. We obtained a similar result regarding the evaluation of these tools: only previous knowledge was related to a higher evaluation, while the size and number of projects were independent. This result suggests that these tools can be applied in different contexts, i.e. a few large projects or several smaller ones, with the same level of satisfaction. Furthermore, they highlight the importance of training and documentation in developing the requisite knowledge and formalize an impact assessment methodology within the organization. Developing specific guidelines based on standard methodologies can, in fact, help the organization to establish impact assessment as a formal phase and help all the stakeholders to develop a common understanding of the scope of the project.

Moreover, to further encourage their adoption, it would be important to raise awareness of the PCM and LF tools also among donors, so that they require their adoption by NGOs. In parallel, it would be important to standardize such methodologies so that an NGO can use the same approach for all its projects and donors. At the same time, flexibility should be allowed because these tools can be further improved and project managers should be free to adapt them to their needs and to the specificities of each project.

Finally, despite the widespread adoption of such tools, it should be stressed that, on average, the project managers surveyed found PCM and the LF quite useful in terms of (social) impacts of the projects, but not particularly easy to use. Moreover, 77 project managers out of 92 (i.e. 84%) only picked and adapted tools belonging to PCM rather than using the methodology ‘as is’. These results again suggest the importance of training and standardization, but also the importance of further development of the methodologies. Considering the limitations highlighted in the literature (e.g. Couillard et al. 2009), improvements to these tools seem necessary, and they could represent interesting avenues for further research.

These considerations are further strengthened by one limitation of our study: we managed to involve a significant number of project managers in the survey, but our sample may have had a small sample bias. Despite our efforts to present the survey as an anonymous and general study on project management practices in ID projects, some project managers may have decided not to answer because they were aware of possessing limited knowledge of the general and specific methodologies. In light of the sample controls that we performed on non-respondent project managers, we believe that this effect was rather limited. Furthermore, we believe that its presence further strengthens the main result of our study: the percentage of adopters of the general methodologies and of the specific tools could have been even lower, thus again highlighting the need for training and improvement of the methodologies.

Finally, to expand our work further, it would be useful to conduct interviews with the project managers to receive feedback on improvements in the tools available. Moreover, although we do not think that Italian NGOs significantly differ from those of other developed countries, it would be interesting to extend our survey to other countries in order to verify the results, deepen the analyses, and expand them to include other topics. In particular, it could be interesting to analyze whether there are cultural and technical differences between the approaches used and the tools adopted in Latin countries (Italy, Spain, etc.), Anglo-Saxons countries (USA, UK, etc.) and northern European countries.

In general, given the importance and specificities of these projects, further research is necessary (Youker 2003; Khang & Moe 2008). In particular, it would be interesting to determine the limitations and areas for improvement of these tools in order to support project performance and evaluation. A wider adoption of these tools could improve the appraisal process thanks to the promotion of a systematic collection of verifiable and comprehensive data and information, universally recognized as an essential starting point for any good evaluation (Morrison-Saunders & Sadler 2010). The systematic collection of data promoted by these tools could radically improve project results and impacts, not only in terms of time, cost, and quality, but also in terms of stakeholder involvement and efficacy in addressing local communities’ problems.

Disclosure statement

No potential conflict of interest was reported by the authors.