Using data differently and using different data

ABSTRACT

The lack of adequate measures is often an impediment to robust policy evaluation. We discuss three approaches to measurement and data usage that have the potential to improve the way we conduct impact evaluations. First, the creation of new measures, when no adequate ones are available. Second, the use of multiple measures when a single one is not appropriate. And third, the use of machine learning algorithms to evaluate and understand programme impacts. We motivate the relevance of each of the categories by providing examples where they have proved useful in the past. We discuss the challenges and risks involved in each strategy and conclude with an outline of promising directions for future work.

KEYWORDS:

• Data
• measurement
• impact evaluation
• machine learning

Acknowledgements

The authors would like to thank the CEDIL board of referees for their comments on earlier versions of this paper. This work was supported by the UK Department for International Development, under Grant 203569.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. An example of how to incorporate measures of parental expectations on a child’s human capital accumulation over the life cycle can be found in Attanasio et al. (2017).

2. These measures are often designed at the level of the agent, but other, higher-level variables such as district or village level observable characteristics may also prove helpful.

3. These adjustments are discussed more at length in Section 5.

4. For the purposes of this paper, ML will refer broadly to the design of algorithms that optimise certain quantitative tasks, and draw heavily from the fields of statistics and computer science.

5. The different nature of these two statistical tasks was first described by Leo Breiman, in his comparison of the data modelling culture, associated with econometrics, and the algorithmic modelling culture, more prevalent in data science (Breiman 2001).

6. Arribas-Del, Patino, and Duque (2017) use similar input to predict an indicator of living environment deprivation for the city of Liverpool.

7. As expected, results from LASSO covariate selection methods vary according to the value of the penalty parameter λ. Several methods have been proposed for the selection of an optimal λ, such as cross validation (James et al. 2013).

8. In another recent study, Bloniarz et al. (2016) propose a different LASSO-based selection method and show that, even in an experimental context, where random assignment to treatment and control groups was largely successful, increases in the accuracy of ATE estimates is achieved. Farrel (2015) builds on the work by Belloni, Chernozhukov, and Hansen (2014) and proposes ‘doubly-robust’ estimator that allows for multi-valued treatments and imposes weaker assumptions on the underlying data generating process.

9. A reliable and not excessively conservative method for correcting standard errors for multiple hypothesis testing can be found in Romano and Wolf (2005).

Additional information

Funding

This work was supported by the Department for International Development [203569].

Notes on contributors

Ingvild Almås

Ingvild Almås is a Professor at the Department of Economics, Norwegian School of Economics (NHH), and Principal Investigator at the Centre of Excellence FAIR (Centre for Experimental Research on Fairness, Inequality and Rationality). Almås is currently a visiting associate professor at IIES, Stockholm University.

Orazio Attanasio

Orazio Attanasio is the Research Director of IFS, Co-Director of the Centre for the Microeconomic Analysis of Public Policy (CPP) and co-directs the Centre for the Evaluation of Development Policies (EDePo). He is a Professor at University College London, a Research Associate at the National Bureau of Economic Research (NBER), a Senior Fellow at the Bureau for Research and Economic Analysis of Development and a Research Fellow at the Centre for Economic and Policy Research. He is currently President of the European Economic Association, and is a member of the Council of the Royal Economic Society.

Jyotsna Jalan

Jyotsna Jalan is a Professor of Economics at the Centre for Studies in Social Sciences, Calcutta in India. She has previously worked the research department of the World Bank in Washington DC and as an Associate Professor of Economics at the Indian Statistical Institute in New Delhi, India.

Francisco Oteiza

Francisco Oteiza is a Senior Research Associate at the CLOSER Institute, based in the Institute of Education, University College London. During the writing of this paper, he worked in the development sector of the Institute for Fiscal Studies, the Centre for the Evaluation of Development Policies (EDePo). He holds a PhD from Royal Holloway, University of London.

Marcella Vigneri

Marcella Vigneri is a Research Fellow with the Center of Excellence in Development Impact and Learning at the London School of Hygiene & Tropical Medicine. She has collaborated for over 10 years with the Ghana Strategy Support Programme, IFPRI, and has previously worked for Oxfam GB, the Overseas Development Institute, the Food and Agriculture Organization, and Oxford University