Strengthening ICT4D evaluation: lessons from the fields of program evaluation, IS/IT evaluation, and aid/development evaluation

ABSTRACT

This study suggests ways to strengthen and clarify conceptual underpinnings of information and communication technology for development (ICT4D) evaluation by exploring its associations with other related evaluation fields – program evaluation, information system/information technology (IS/IT) evaluation, and aid/development evaluation. These fields have developed long and rich theoretical discourses, models, and approaches over the years, which are of interest to strengthen evaluation practices in ICT4D. Through detailed content analyses and expert interviews, we find that ICT4D evaluation shows significant parallels with the three fields in theory and practice. Also, we see that ICT4D evaluation has developed its own discourse – in particular, ICT4D researchers have valued capturing the situated development context of ICT. We argue that ICT4D evaluation can learn more from other evaluation traditions and disciplines, in order to help move the field to a more ‘mature’ stage.

KEYWORDS:

• ICT4D evaluation
• program evaluation
• IS/IT evaluation
• aid/development evaluation

Acknowledgements

The article was based on the first author’s doctoral dissertation work (Yim, 2019). The second author was the dissertation supervisor of the first author. The links to the pages regarding the dissertation: (1) ProQuest: https://search.proquest.com/dissertations-theses/information-communication-technology-development/docview/2317590200/se-2?accountid=27828 (2) University of Washington ResearchWorks: https://digital.lib.washington.edu/researchworks/handle/1773/44837

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 This paper uses the terms ‘ICT(s) for development (ICT4D)’ and ‘ICT(s) and development (ICTD)’ interchangeably. Yet, the paper also respects views that distinguish the two terms. For example, Dearden and Tucker (2016, p. 37) explain that ICT4D indicates ‘devising technologies and establishing (socio-)technical interventions to contribute toward development’ whereas ICTD means ‘studies of the processes and consequences of technology adoption’. Therefore, whenever this paper referred to other works, it tried to consistently use the term mentioned in the original work.

2 This study follows the understanding that ‘a program’ consists of smaller ‘projects’ (Cook et al., 1985 cited in Shadish et al., 1991). Throughout this paper, ‘ICT4D projects’ indicate ‘ICT4D projects and programs.’

3 In 2019, the criteria were updated with revised definition for each criterion and an addition of ‘coherence’ criterion assessing ‘[t]he compatibility of the intervention with other interventions in a country, sector or institution.’ (OECD/DAC Network on Development Evaluation, 2019, p. 8). Yet, considering that this is a recent revision that took place after the data collection and analyses were completed for this study, and that the influence of the previous five criteria has been significant in the field over several years, we refer to the previous version of the criteria.

4 In choosing number of journals to be reviewed, the study placed a greater weight on ICT4D journals (three journals) compared to program evaluation, IS/IT, and development journals (one from each field). The decision was made based on the assumption that ICT4D journals would have a greater number of articles on the specific topic that the study seeks to analyze (i.e. journal articles on evaluating ICT4D projects) compared to the journals of the other three fields. Moreover, due to the constraint of the study period, it was practically difficult to review and select articles from more than six journals.

5 2000 was chosen as the starting year because it is the year since when ICT has been actively promoted and recognized as a tool for development – e.g. the establishment of the Digital Opportunity Task Force at the G8 Summit (2000), World Summit on the Information Society (WSIS) in Geneva (2003) and Tunis (2005), etc. (Gomez, 2013) Accordingly, the assumption was that from 2000 and onwards, there would have been a significant number articles (and reports) published on ICT4D projects and their evaluation.

6 For more detailed explanation of inclusion/exclusion criteria in selecting journal articles and aid/development agency reports for content analyses, please see the Appendix.

7 Whenever it says ‘(multiple codes possible)’ in the figure or table title, it means that multiple different codes at the same level may have been applied to a single reviewed work.

8 During the interview process, it was realized that there is only a blurred line between academics and practitioners in the ICT4D field, e.g. a person can be both an academic and a practitioner in the field. Perhaps this was due to the field’s heavy practical focus. Therefore, it was decided that rather than comparing responses according to the strict occupational dichotomy of academics vs. practitioners, it was examined how interviewees’ responses reflect similar or different perspectives from academic vs. practitioner spheres.

9 For tables in Section 4.2, the following applies: original sources mostly identified and re-cited from the reviewed works; excluded those cases where evaluation model or framework could not be fully understood, due to reasons e.g. not enough explanation in the article about the model or framework and unable to access the original source.

10 As mentioned in Section 3, an evaluation model, theory or framework was categorized under ‘ICT4D evaluation’ when it illustrates a combination of two or more disciplines – among program evaluation, IS/IT evaluation, aid/development evaluation, and/or other discipline(s) – and/or when it primarily stems from ICT4D discourse.

11 According to Rogers (2014), theory of change illustrates the expected process of how an intervention’s activities lead to results that in turn contribute to achieving intended impacts. It usually entails identifying input, output, outcome, and impact elements and deriving and exploring causal connections among them.

Additional information

Funding

The first author’s doctoral dissertation work, based on which this paper was written, was supported by the University of Washington Information School from the PhD Dissertation Fund.

Notes on contributors

Moonjung Yim

Dr Moonjung Yim received her PhD in Information Science at the University of Washington Information School. Her research interest is on examining the use of information and communication technology (ICT) and changes in individuals and society, and on evaluating ICT for development (ICT4D) projects or programs. Other areas of interest include program evaluation, information system/information technology (IS/IT) evaluation, and aid/development evaluation. She currently works as a Post-doctoral Researcher at Korea Advanced Institute of Science and Technology (KAIST)’s Korea Policy Center for the Fourth Industrial Revolution (KPC4IR).

Ricardo Gomez

Dr Ricardo Gomez is Associate Professor at the University of Washington Information School, and faculty affiliate with the Latin American & Caribbean Studies Program, the Harry Bridges Center for Labor Studies, and the UW Center for Human Rights. His research interests focus on the uses of information and communication technologies in international and community development contexts, and on migration, human rights and social justice. He specializes in social dimensions of the use (or non-use) of communication technologies, and how they contribute to well-being. He is particularly interested in qualitative research methods and in group facilitation and process design. These methods help him find creative ways to communicate complex ideas and research results in everyday language.