Evaluation of Information Systems Project Success – Insights from Practitioners

Figures & data

Figure 1. Research framework and research questions.

Table 1. Profile of respondents (project managers and companies)

Profile of project	Frequency	Per cent		Profile of companies	Frequency	Per cent
managers				Total employees
Gender				< 201	24	37.50
Male	53	82.81		201– 500	15	23.44
Female	11	17.19		501– 2000	10	15.62
Age				> 2000	15	23.44
< 31	6	9.38		International presence
31– 40	26	40.62		Yes	53	82.81
> 40	32	50.00		No	11	17.19
Training or certification in project management				Certifications
Yes	34	53.13		Yes	31	48.44
No	30	46.87		No	32	50.00
Average years in project management				Do not know/Do not answer	1	1.56
1– 10	35	54.69		Project management approach/methodology
11– 20	28	43.75		PMBOK or Custom (based on PMBOK)	24	37.50
> 20	1	1.56		Custom (based on various methodologies)	14	21.87
Number of projects as a project manager				Custom	6	9.38
< 11	23	35.94		It is not used a formal methodology	16	25.00
11– 20	13	20.31		Other	4	6.25
21– 30	8	12.50		Uses a project management maturity model
> 30	20	31.25		Yes	10	15.62
				No	54	84.38

Table 2. Activity sector of companies

Active sector	Frequency	Per cent	Accumulated per cent
Engineering	16	25.00%	25.00%
Public administration	6	9.37%	34.37%
Education	5	7.81%	42.18%
Finance, insurance and banking	5	7.81%	49.99%
Manufacturing	3	4.69%	54.68%
Electricity, water and gas	2	3.13%	57.81%
Communication and transportation	2	3.13%	60.94%
Others	25	39.06%	100.00%
Total	64	100.00%

Table 3. Definition of the process of evaluating the success of a project

Process of evaluating the success of a project	Frequency	Per cent	Accumulated per cent
Defined before project initiation (general, project independent)	3	4.69	4.69
Defined at project initiation phase (before initial planning is carried out)	14	21.88	26.57
Defined at the planning (initial) phase of the project	11	17.19	43.76
Defined at project executing phase	4	6.25	50.01
Defined at project closing phase	4	6.25	56.26
Defined as needed, at any time during the project	4	6.25	62.51
Defined at the delivery of products/services	1	1.56	64.07
Not formally defined	23	35.93	100.00

Table 4. Definition of criteria to evaluate success

Success criteria	Frequency	Per cent	Accumulated per cent
There is no formal definition of criteria for success assessment.	28	43.75	43.75
There is a predefined list of criteria, which are discussed and complemented with the participation of several stakeholders.	20	31.25	75.00
The evaluation criteria are defined together with stakeholders in each project.	13	20.31	95.31
There is a predefined list of criteria, which is not discussed with stakeholders.	3	4.69	100.00

Table 5. Criteria formally used to evaluate project success

Criteria	1st to 11th			Criteria	12th to 22th
	Number of responses	Per cent			Number of responses	Per cent
1. Time compliance	39	60.94		12. Use of IS solutions by the customer	7	10.94
2. Scope compliance	39	60.94		13. Intangible benefits	6	9.38
3. Cost compliance	35	54.69		14. Preparation for the future	4	6.25
4. Client/customer satisfaction	32	50.00		15. Personal development of team members	4	6.25
5. Compliance with the business goals set for the project	22	34.38		16. Public recognition of the project	4	6.25
6. Quality of deliverables	22	34.38		17. Economic impact	3	4.69
7. Compliance with the client/customer’s business objectives	12	18.75		18. Relationship between stakeholders	2	3.13
8. User satisfaction	12	18.75		19. Compliance with the vendors’ business objectives	1	1.56
9. Executing team satisfaction	9	14.06		20. Other stakeholders satisfaction	1	1.56
10. Sponsor satisfaction	9	14.06		21. Social impact	1	1.56
11. Contribution to the development of the organization	9	14.06		22. Environmental impact	1	1.56

Table 6. Participants in the evaluation process and definition of criteria to evaluate success

Participants	Definition of criteria			Evaluation process
	Number of responses	Per cent		Number of responses	Per cent
Project Manager	46	71.88		44	68.75
Client	21	32.81		29	45.31
PMO	21	32.81		18	28.13
Sponsor	20	31.25		28	43.75
Management team	18	28.13		22	34.38
Executing team	14	21.88		23	35.94
There are no participants	10	15.63		9	14.06
Other participants	5	7.81		3	4.69
External consultants	1	1.56		1	1.56

Table 7. When success is evaluated

Phase	Number of responses	Per cent
Project planning	3	4.69
Project execution, one time	1	1.56
Project execution, several times	20	31.25
Project closing	42	65.63
The project success is not evaluated formally	15	23.44
Other	1	1.56

Table 8. Ways/tools used to obtain information to evaluate the success

Ways/tools for data collection	Number of responses	Per cent
Reports	29	45.31
Meetings	29	45.31
Deliverables	23	35.94
Surveys	17	26.56
There are not used any means for evaluating the success formally	16	25.00
Interviews	7	10.94

Table 9. Information about the last completed projects

Type of project	Frequency	Per cent	Project budget (€)	Frequency	Per cent	Accumulated per cent
Software development	70	44.30	< 25,001	42	28.38	28.38
Package implementation	20	12.66	25,001–50000	23	15.54	43.92
Infrastructure implementation	19	12.03	50,001–100000	18	12.16	56.08
Systems migration	15	9.49	100,001–500000	37	25.00	81.08
Consultancy and business analysis assignments	14	8.86	> 50,000	28	18.92	100.00
System enhancement	10	6.33	Duration of project (month)	Frequency	Per cent	Accumulated per cent
Others	6	3.80	1 a 3	52	32.91	32.91
Outsourcing e insourcing	3	1.90	4 a 6	34	21.52	54.43
Systems integration	1	0.63	7 a 12	47	29.75	84.18
Total	158	100.00	> 12	25	15.82	100.00

Table 10. The success achieved in projects

Scale of success	Number of responses	Per cent	Accumulated per cent
0 – Abandonment of the project
3	1	0.63	0.63
5	5	3.16	3.80
6	18	11.39	15.19
7	33	20.89	36.08
8	57	36.08	72.15
9	27	17.09	89.24
10 – Total success	17	10.76	100.00
Total	158	100.00

Table 11. The opinion of the participants regarding the evaluation success aspects

Opinion	Participants
“There should be a formal project evaluation process. Given the low degree of maturity in the company in these processes, I suggested starting with an evaluation after the end of the project, based on checklists for internal evaluation (to be completed by the project manager), internal questionnaire (to be completed by the project team) and questionnaire plus an interview with client and sponsor.”	4
“The business value delivered (by the solution) should be assessed.”	9, 10
“It would be useful to survey customers and users, as well as other stakeholders.”	12
“The criteria of cost, time, scope, quality of products/services, and impact on the organization’s activity, should be used in our projects.”	14
“In the evaluation process should be considered quality, planning, and safety.”	15
“The evaluation process should be defined in the setup of the project”; “Evaluation criteria should be defined when defining the project.”	22, 46
“(It would be important) to adopt a formal method for managing and evaluating success”; “A formal evaluation process should be adopted (provided that it is sponsored by management).”	24, 26, 40, 41
“It would be important to collect and analyze feedback from members of the project’s execution team, as well as sharing and analyzing lessons learned.”	42
“Formalize the quantification and assessment of project success.”	54
“Carry out satisfaction surveys to the project team and client areas.”	55
“EVM should be implemented. Success assessment should be aligned with the business model. The assessment should be adapted to each case.”	56
“Define KPI (Key Performance Indicators) and make an analysis of quantitative and qualitative results.”	61
“Define compliance targets with sponsor participation and use it to assess success.”	64

Figure 2. Summary of main insights from IS project success management practice.