ICT4D evaluation: its foci, challenges, gaps, limitations, and possible approaches for improvement

ABSTRACT

With significant cases of failures in information and communication technologies for development (ICT4D or ICTD), much attention has been paid to the importance of evaluation to improve project sustainability. Using the Capability Approach and human development notion as a theoretical lens, this study examines the foci of the ICT4D evaluation, its challenges and gaps, and possible approaches to address the challenges and gaps. The study conducted content analyses of 108 peer-reviewed journal articles and 114 aid/development agency reports, and interviews with 24 researchers in the academic and practitioner spheres. The study finds that the foci of the ICT4D evaluation can be described in terms of areas such as continental/regional focus, method and research design, notion of development, and evaluation focus. Moreover, the study finds that challenges exist surrounding impact assessment, which can be addressed by clarifying the development notion–impact connection in the evaluation process and focusing on the evaluation capacity building (ECB) of project participants. In addition, the study highlights the importance of the domain-based approach in the ICT4D evaluation.

KEYWORDS:

• ICT4D
• evaluation
• impact
• domain-based approach
• Capability Approach

Acknowledgments

The article was based on the first author’s doctoral dissertation work (Yim, 2019). The second author was the dissertation supervisor of the first author. The links to the pages regarding the dissertation: (1) ProQuest: https://search.proquest.com/dissertations-theses/information-communication-technology-development/docview/2317590200/se-2?accountid=27828 (2) University of Washington ResearchWorks: https://digital.lib.washington.edu/researchworks/handle/1773/44837

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 This study agrees with the interpretation that ‘a program’ is a larger unit than ‘a project’, and that a program can be composed of several different projects (Cook et al., 1985 cited in Shadish et al., 1991). Hereafter, this study will use the phrase ‘ICT4D projects’ to indicate ‘ICT4D projects and/or programs.’

2 Available as of February 2018.

3 EvalNet is a network of evaluation managers and specialists from ‘development co-operation ministries and agencies in OECD-DAC member countries and multilateral development institutions’ with the aim of sharing lessons, knowledge, and practices in development evaluation (Organisation for Economic Co-operation and Development (OECD), 2015).

4 ‘(multiple codes possible)’ on the table title indicates that a single article or report could have been coded multiple times. For example, a report written jointly by multiple aid/development agencies or related ministries would have been coded more than once.

5 Although an interviewee was only involved in ICT4D project evaluation once, he/she was included in the interviewee pool, as he/she was one of the authors of an analyzed report, and the evaluation design and implementation were conducted for multiple years.

6 The ‘learning’ type was excluded because this study excluded articles and reports that addressed requirements or conditions to use ICTs (e.g. assessing the level of ICT skills). Instead, the study selected those that deal with evaluating ICT4D projects or programs.

Additional information

Funding

The first author’s doctoral dissertation work, based on which this paper was written, was supported by the University of Washington Information School from the PhD Dissertation Fund.

Notes on contributors

Moonjung Yim

Dr Moonjung Yim received her PhD in Information Science at the University of Washington Information School. Her research interest is on examining the use of information and communication technology (ICT) and changes in individuals and society, and on evaluating ICT for development (ICT4D) projects or programs. Other areas of interest include program evaluation, information system/information technology (IS/IT) evaluation, and aid/development evaluation. She currently works as a Post-doctoral Researcher at Korea Advanced Institute of Science and Technology (KAIST)’s Korea Policy Center for the Fourth Industrial Revolution (KPC4IR).

Ricardo Gomez

Dr Ricardo Gomez is Associate Professor at the University of Washington Information School, and faculty affiliate with the Latin American & Caribbean Studies Program, the Harry Bridges Center for Labor Studies, and the UW Center for Human Rights. His research interests focus on the uses of information and communication technologies in international and community development contexts, and on migration, human rights and social justice. He specializes in social dimensions of the use (or non-use) of communication technologies, and how they contribute to well-being. He is particularly interested in qualitative research methods and in group facilitation and process design. These methods help him find creative ways to communicate complex ideas and research results in everyday language.