On international household survey data availability for assessing pre-pandemic monetary and multidimensional poverty in developing countries

ABSTRACT

Data availability plays a crucial role in the fight against poverty. Yet, it lags behind the data available on most other economic phenomena. This paper catalogs and reviews existing data availability for low- and middle-income countries with a view to break the cycle of outdated poverty data and strengthen statistical systems – while drawing readers’ attention to existing information and experiences. Countries that generate and analyze frequent and accurate poverty data are identified to show what is possible and to better document what is already available. Results show that data for both monetary and multidimensional poverty dramatically increased since 1980. Sixty countries already produce annual updates to key statistics, and some have continuous household surveys with cost-cutting synergies. International agencies have explored short surveys for comparable data but the success and uptake of these have not followed expected patterns. Certain regions have agreed on harmonized variable definitions across countries, and new technologies reduce lags between data collection and analysis. These existing resources and experiences can inform much-needed efforts to expand data availability.

KEYWORDS:

• Poverty measurement
• data availability
• household surveys
• development

JEL:

• I32, O15, D31, C80

Introduction

The COVID-19 pandemic has created a setback in global poverty trends – but where are the data to probe these trends? Data on poverty are severely limited both in terms of frequency and coverage. The limitation in frequency is especially striking when compared to the data availability concerning other economic phenomena. GNI data are published annually,¹ while inflation and external debt statistics are available on a quarterly basis. Stock market data are made public every day, and with the invention of high-frequency trading, it has become available for investors at the fraction of a second. Dissatisfied with this disparity, the post-2015 development agenda identified the need for regularly updated data to monitor Sustainable Development Goals (SDGs), the first of which is ending poverty in all its forms. In the context of the pandemic, which set a landmark in the data landscape, this paper exhaustively catalogs and reviews the poverty data from international multi-topic surveys that could be used for poverty measurement and analyses, from their inception through early 2020 – that is, up until the brink of the pandemic. While many energetic efforts have been initiated during the pandemic, the data organization in this paper fills an ongoing gap to landscape existing data that could illuminate poverty trends, and interlinkages across indicators, across developing countries.

In using the term poverty in this paper, we signify both monetary and multidimensional poverty.² For example, the $1.25/day poverty measure reflected money poverty and was published for 115 countries using data 2000–2012; the $1.90 is published for 118 developing countries 2000–2012.³ The global Multidimensional Poverty Index⁴ complements the $1.90/day measure by measuring multidimensional poverty and has been published for over 120 countries. In an open letter⁵ to the High Level Panel advising the United Nations on the content of a post-2015 development agenda, more than 120 Southern non-governmental organizations stated their number one concern was that ‘Poverty is multidimensional and should not be narrowly defined and measured only as a matter of income.’ The final Sustainable Development Goals’ first two targets under the first goal include (a) a target of eradicating $1.25/day poverty and (b) a target focused on ‘poverty in its many dimensions’. The data requirements to monitor progress in poverty in several dimensions are the focal issue of concern in this paper.

Nearly every country in the world uses household surveys to produce its poverty statistics, whether these are income or consumption poverty, or multidimensional poverty. Thus, in this paper, we limit the scope to poverty data based on household surveys; elsewhere we have considered insights that other data sources can contribute (Alkire and Samman 2014).

In spite of the explosion of economic data availability, many authors studying various dimensions of poverty have brought to light data limitations.⁶ In terms of frequency, poverty data continues to lag behind other economic information, as it is often collected only every three to ten years – and often published a full year or two after data collection is finished. In terms of coverage, by and large most poverty datasets still miss information on important dimensions of poverty such as violence, empowerment or informal work – as well as key indicators such as quality of services (Alkire 2007). But on the other hand, some available data are only scantily analyzed. For example, even when multiple poverty indicators are available from the same survey, most indicators are analyzed in a dashboard style, ignoring how many multiple interconnected deprivations individual people or households experience, and consequently providing little information for integrated, cross-cutting or coordinated policy responses.

This situation does not meet the demands of policy. Managing initiatives that reduce poverty requires timely data to plan, monitor, evaluate, and re-design policies. Management requires recent data that are cleaned and analyzed promptly – and analyses that provide information in the form required for policy coordination and integrated multi-sectoral response.

Despite the limitations of currently available data, there are more poverty data for developing countries now than in any previous period in history. For example, this paper identifies 121 developing countries with monetary poverty data and 137 countries with multi-topic household survey data. Further, the content of that data has expanded significantly, including data from the same survey, and the patterns of its expansion seem to be catalyzed in part by data needs of the MDGs (Cassidy 2014) and SDGs.

The aim to increase the periodicity and timeliness of household surveys is longstanding. Attempts at innovations have had mixed results, yet these experiences – both negative and positive – are illuminating.⁷ This paper traces recent developments in certain household surveys, showing their tremendous rise since the 1980s, yet observing that the gaps in poverty data remain a key constraint in the fight against poverty. It then describes national annual surveys including some which are both nationally produced and create comparable indicators. It also discusses shortened surveys (KIS, Interim DHS and CWIQ) promoted by international agencies, and closes with examples of how time-saving survey technologies support data collection at decreased cost. Taken together these examples shed some light on the question of whether a step-change in the generation of poverty data, and its effective use to eradicate poverty, might come to pass – and if so, what avenues might be pursued.

Existing poverty data: level and trends

Poverty data for developing countries have made huge leaps in the last thirty years.⁸ There are more data now than in any previous period in history. Further, the content of that data has expanded significantly, with the patterns of its expansion fueled by widened national priorities and capabilities and also by international interest in topics including the SDGs. Surveys are just one source of poverty data. Many countries have data for key SDG indicators from multiple sources: census data; survey data (both national survey data and international, i.e. from DHS, MICS, CWIQ and LSMS) and administrative data. There is also an active exploration of the potential of ‘big data’ to improve sampling frames and to provide poverty-relevant indicators, such as electricity and road access.⁹

Here, we document the dramatic rise in poverty-related household surveys in developing countries since 1980.¹⁰ The good news of this rise is certainly to be celebrated. We track the surveys that have been completed, and which have issued reports. A great (and desirable) degree of data availability occurs in circumstances in which the micro-data are available. Micro-data are publicly available, or available upon request for some of the surveys included (most DHS and MICS), but not others.

While such a review could include many survey forms including labor force surveys or those fielded in OECD countries, we focus here on the rise of household surveys in developing countries that can be used to analyze monetary poverty or that address at least three dimensions related to multidimensional poverty. We focus on two periods: 1980–2020 in the case of monetary poverty data, and 1985–2020 for multidimensional poverty data.

Household surveys for monetary poverty, 1980–2020

The precise number of available household surveys that are exclusively or partially concerned with household income or consumption and expenditure is hard to determine since a myriad of online search engines and survey networks currently exist. They include poverty data that is collected at different moments in time, on disparate administrative levels and using divergent data gathering methods. To illuminate the field while seeking data quality, we have restricted the analysis of monetary household surveys to those listed on the main page of PovcalNet, the World Bank’s regional survey aggregation website.¹¹

As the left panel of Figure 1 indicates, the absolute number of income or consumption and expenditures surveys, as well as the absolute number of countries with such monetary surveys, have dramatically increased from the early 1980s until 2020.¹² By the procedures followed in the study, survey data on income or consumption and expenditure are available for 121 countries.¹³

Figure 1. Monetary surveys; cumulative and number per year.

The right panel of Figure 1 shows the number of ‘new’ surveys fielded each year and number of ‘new’ countries gaining surveys each year. These marginal increases were greatest during the late 1980s and the mid-1990s respectively.

In total, 1,033 monetary surveys are listed during the period 1980–2020. The country with the highest number of surveys in this period is Brazil (35), followed by Costa Rica (33), Romania (33), and Honduras (31).

Household surveys for multidimensional poverty, 1985–2020

Many surveys are fielded which collect SDG-related or deprivation-related information related to services, but not necessarily on monetary poverty. Due to restrictions with regards to information on data coherence, quality, and availability, a comprehensive overview of all existing national multidimensional household poverty surveys cannot be provided. There is no equivalent to PovcalNet for multidimensional poverty datasets.

For the purposes of this paper, we have simply identified seven major multidimensional surveys for quantitative analysis and listed their trajectory since 1985 (the earliest date of surveys). Each of these surveys fulfills the following three criteria: (1) The survey must measure at least three aspects of well-being (such as health and nutrition, education, living conditions, or employment); (2) The survey must be relevant for the comparative study of [some] developing countries; (3) The survey must be widely used and provide high-quality data.

Four surveys to which these criteria apply are the Demographic and Health Surveys (DHS), which collects data on population, health, HIV, and nutrition; the Core Welfare Indicator Questionnaire surveys (CWIQ) which collect information on household well-being and basic community services; the Multiple Indicator Cluster Surveys (MICS) which monitor the situation of women and children, particularly with regards to health and education. The Living Standards Measurement Study (LSMS) office of the World Bank provides technical assistance to many surveys that are not listed as LSMS; we include LSMS surveys listed on their website which measure consumption behavior, economic well-being, and a variety of sectoral aspects such as housing, education and health.¹⁴ We also include World Health Surveys (WHS) alongside surveys listed in International Household Survey Network (IHSN) as Integrated Survey (non-LSMS) or Integrated Living Conditions Survey (ILCS).

Together these contribute 992 surveys. Just as the monetary surveys included income or consumption based on various definitions, so do not the surveys reported here all contain the same indicators or definitions. The number of each kind of survey, and country coverage, appear in Table 1; a list by country appears in Appendix A.

Table 1. Major multi-topic surveys.

Survey	Number of surveys	Countries covered	Website
DHS	409	90	https://dhsprogram.com/What-We-Do/survey-search.cfm
MICS	284	107	http://www.childinfo.org/mics_available.html
LSMS	132	36	http://microdata.worldbank.org/index.php/catalog/lsms
CWIQ	51	24	http://catalog.ihsn.org/index.php/catalog
ILCS or IS	73	12	http://catalog.ihsn.org/index.php/catalog
WHS	43	43	http://apps.who.int/healthinfo/systems/surveydata/index.php/catalog/whs

It must be noted that these seven surveys do not include the extensive multi-topic household surveys that have been completed at national levels to investigate quality of life, social indicators, and living conditions. To create a catalogue of multi-topic surveys that include national surveys it would be necessary to construct the relevant criteria and apply these to multiple data banks. Appendix B introduces 14 data portals that might be consulted for such a task, as well as a series of datasets organized by region.

The left panel of Figure 2 shows that even using just this cross-section of surveys, the number of multidimensional household poverty surveys has increased dramatically since 1985 and now covers 137 countries. As we see from the right panel, major increases of both multidimensional surveys and the countries with multidimensional surveys occurred during the mid-1990s, 2000, 2005, and 2010 – corresponding with the rollout of successive phases of the MICS surveys. Of the 992 surveys listed here, Malawi (34) and Tanzania (31) have the highest number of surveys.

Figure 2. Multidimensional surveys; cumulative and number per year.

From this brief and incomplete review, we can nonetheless observe the strong rise in data availability for both monetary and multidimensional poverty since the 1980s. The strong gains since the 1980s, and the increase in pace since 2000, all show that household surveys have not at all been static. But has this salutary progress been sufficient? The resounding consensus is that it is not.

Ongoing limitations: content, quality, frequency, timeliness and availability

Existing data on poverty remains limited – particularly in terms of content as they overlook key indicators, data quality which is variable; frequency of surveys, timeliness of data publication and analysis, and availability of that data.

A thorough review of these issues is not presented here, for many have already identified them in depth and the Data Revolution, which the High Level Panel summoned, has caught the imagination of many. This section simply reminds the readers of the points made in a myriad of studies.

In terms of frequency, as mentioned above, poverty data tend to be relatively infrequent. Coverage is restricted as poverty data still misses information on important dimensions of poverty such as violence, empowerment or informal work. Even information on basic variables like health remains severely limited. Also, most poverty analysis does not address the interconnectedness of deprivations that lock people into poverty. The first key message in The MDGs at Mid-point – a 50-country study on accelerating progress that the UNDP released in 2010 – was that successful countries had addressed different deprivations together because of these interconnections. The joint distribution of deprivations – which can be seen using multi-topic surveys – can be analyzed to inform joined-up policies – through multidimensional analyses.

Many examples have been used to show the scale of the problem. For example, India had no publicly available and nationally representative official multidimensional micro-data between 2005/6 and 2015/16. SDG assessments of data availability observed severe gaps in the ability of most countries to report trend data on many SDG indicators. This is not new: in the previous era, the mid-point assessment of the MDGs led by an eminent group of economists observed that:

Many, among the poorest and most vulnerable countries, do not report any data on most MDGs. When it is available, data are often plagued with comparability problems, and MDG indicators often come with considerable time lags. Improving data gathering and its quality in all countries should be a central focus of the second half of the MDG time frame and beyond. Reliable data and indicators are essential, not only to enable the international development community to follow progress on MDGs, but also for individual countries to effectively manage their development strategies. Bourguignon et al. (2008, 6)

While efforts to improve poverty data have increased the content and frequency of poverty data, attention is drawn to ongoing data gaps again and again, for example in the final report on the MDGs:

Despite considerable advancements in recent years, reliable statistics for monitoring development remain inadequate in many countries. Data gaps, data quality, compliance with methodological standards and non-availability of disaggregated data are among the major challenges to MDG monitoring. United Nations (2014)

Notwithstanding the visible lack of poverty data, in some cases, funds are invested in multi-topic household surveys that are never fully analyzed – perhaps because their content, or the timeliness of data cleaning and publication, did not meet policy needs. The issue of data creation and its use for policy must thus be considered together.

Experiences in annual multi-topic household surveys

The previous section addressed the steep rise in the number of countries having at least one data point, as well as of multiple data points. This section now focuses on experiences in more frequent data collection, reporting, analysis and policy use.

National surveys

Many countries have frequent household survey instruments in place for some core indicators of human poverty.¹⁵ However, there does not seem to be a publicly accessible and complete record of these surveys internationally.¹⁶ Yet despite the perception that annual or biennial data are very rare, we have encountered quite a range of such experiences.

A few countries update a wide range of poverty data regularly. For example, Colombia updates both official income and multidimensional poverty data and statistics annually and Mexico does so every two years. The European Union Statistics on Income and Living Conditions (EU-SILC) described more fully below, provide annual official updates of the EU-2020 multidimensional poverty and social exclusion indicators – covering quasi-joblessness, material deprivation, and being at-risk-of (relative) income poverty – for over 30 countries.

More commonly, the annual surveys either primarily collect monetary poverty data or primarily cover some dimensions of poverty but do not include detailed income or consumption and expenditure modules. For example, India’s National Sample Survey (NSS) provides annual updates of consumption poverty, with a large round for greater disaggregation roughly every five years. Pakistan’s Social and Living Standard Measurement Survey (PSLM) fields annual surveys, alternating between two questionnaires and between district- and province-level disaggregation potentials.

Some countries have moved to higher-than-annual frequency: Indonesia’s SUSENAS collects consumption poverty data every quarter and releases poverty statistics twice per year. Ecuador has a multi-topic survey that provides three nationally representative statistical updates per year, and at lower levels of disaggregation annually.

Table 2 presents an incomplete list of annual surveys that are implemented by national statistics offices. It covers 60 countries and surely excludes some existing experiences.¹⁷

Table 2. Sixty Annual Household Surveys.

1. Argentina (EPH-C)	21. Germany (EU-SILC)	41. Pakistan (Pakistan Social and Living Standards Measurement PSLM)
2. Armenia (Household’s Integrated Living Conditions Survey)	22. Greece (EU-SILC)	42. Panama (Encuesta de Hogares – EH)
3. Austria (EU-SILC)	23. Honduras (Encuesta Permanente de Hogares de Propósitos Múltiples)	43. Paraguay (Encuesta Permanente de Hogares – EPH)
4. Belgium (EU-SILC)	24. Hungary (EU-SILC)	44. Peru (Encuesta Nacional de Hogares – ENAHO)
5. Bolivia (Encuesta de Hogares)	25. Iceland (EU-SILC)	45. Philippines (Annual Poverty Indicators Survey APIS alternating with Family Income and Expenditure Survey FIES)
6. Brazil (Continuous PNAD)	26. India (National Sample Survey)	46. Poland (EU-SILC)
7. Bulgaria (EU-SILC)	27. Indonesia (SUSENAS)	47. Portugal (EU-SILC)
8. Cambodia (Cambodian Socioeconomic Survey – CSES)	28. Ireland (EU-SILC)	48. Romania (EU-SILC)
9. Colombia (Gran Encuesta Integrada de Hogares)	29. Italy (EU-SILC)	49. Slovakia (EU-SILC)
10. Costa Rica (Encuesta Nacional de Hogares)	30. Jamaica (Survey of Living Conditions)	50. Slovenia (EU-SILC)
11. Croatia (EU-SILC)	31. Kazakhstan (Household Budget Survey)	51. South Africa (General Household Survey GHS, Labour Force Survey)
12. Cyprus (EU-SILC)	32. Latvia (EU-SILC)	52. Spain (EU-SILC)
13. Czech Republic (EU-SILC)	33. Lithuania (EU-SILC)	53. Sweden (EU-SILC)
14. Denmark (EU-SILC)	34. Luxembourg (EU-SILC)	54. Switzerland (EU-SILC)
15. Dominican Rep (Encuesta Nacional de Fuerza de Trabajo)	35. Malta (EU-SILC)	55. Turkey (EU-SILC, annual Household Budget Survey HBS)
16. Ecuador (Encuesta de Calidad de Vida)	36. Mauritius (Continuous Multi-Purpose Household Survey)	56. United Kingdom (EU-SILC)
17. El Salvador (Encuesta de Hogares de Propósitos Múltiples)	37. Moldova (Household Budget Survey)	57. United States (National Health Interview Survey)
18. Estonia (EU-SILC)	38. Netherlands (EU-SILC)	58. Uruguay (Encuesta Continua de Hogares – ECH)
19. Finland (EU-SILC)	39. Nigeria (General Household Survey-GHS)	59. Venezuela (Encuesta de Hogares Por Muestreo – EHM)
20. France (EU-SILC)	40. Norway (EU-SILC)	60. West Bank and Gaza (Expenditure and Consumption Survey)

This list does not exhaust relevant cases, and would be much longer, if the period is extended slightly. A number of countries field surveys every two years rather than annually. In addition to Mexico, these include Viet Nam’s Household Living Standard Survey, Nicaragua’s Encuesta Nacional de Hogares sobre Medición de Nivel de Vida, Thailand’s Household SocioEconomic Survey, and Malaysia’s Household Income and Basic Amenities survey, which is fielded twice in five years.

Continuous national household sample surveys

A challenge of data collection is that not all indicators require annual updates. Certain indicators change slowly so require updating only every three to five years. Some indicators require a long and detailed questionnaire, or a different sample design to focus on a particular subgroup. In some cases, if comprehensive data are available occasionally, estimates can be computed based on variables available in shorter interim surveys (as SWIFT, explained below, is doing for consumption poverty). There are also varying needs for disaggregated data. For these reasons, if management capabilities are sufficiently strong, the ideal institutional arrangement for high-frequency data is the ‘continuous’ national household sample survey, which may have a core module of high-frequency indicators, and rotating modules according to the specific indicator needs. They may also schedule regular but distinct surveys (labor force, agricultural, or health surveys for example).

Indonesia, Ecuador, and others countries including Brazil,¹⁸ have what can be called ‘continuous household surveys’ in that the survey teams are in the field more or less continuously with different surveys and modules. When management capacity is adequate, data quality and availability increases in a way that is cost-saving and coordinated. Different surveys are drawn from a master sample, normally can be aggregated for more in-depth disaggregation, and may have a panel element. In addition to these continuous national household surveys, there is also a ‘continuous DHS’ – which has been implemented in Peru and in Senegal.

While annual updates of poverty figures are not yet the norm, these examples demonstrate their feasibility. In addition, evidence from the 2008 financial crisis suggests that these high-frequency surveys were ‘a good means of gauging the expenditure impacts of shocks and even some of the specific coping mechanisms involved’ (Headey and Ecker 2013, 332). However, the national surveys mentioned above are not comparable to one another. Furthermore, they focus primarily on consumption/expenditure or income data and omit many other core indicators of human poverty. We turn now to various initiatives to generate internationally comparable data, and annual data on these other aspects of poverty.

Internationally comparable short surveys

The Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) have increased in prominence due to their quality, quantity, and comparability, their free public availability, as well as the match between these surveys and key MDG indicators. Because of their data quality, they are used in academic research. Corsi et al report: ‘A … systematic review found that 1117 peer-reviewed publications using DHS data have appeared in more than 200 journals, between 1984 and 2010’ (2012, 1607). Yet, because the DHS and MICS are fielded every 3–5 years (DHS on average just over 5 years; MICS every 5 years in the past, but are moving towards every 3 years), and their cleaning and standardization require some time, they are not designed for annual reporting.

This fact has been overtly recognized and acknowledged by these institutions, which have explored various responses. Their responses are relevant to present discussions. For example, due to the length of the DHS, the DHS office set up the Key Indicator Survey (KIS)¹⁹ whose purpose was to monitor key health and population indicators at a lower level of disaggregation, e.g. districts. KIS questionnaires are ‘designed to be short and relatively simple, but also to be able to produce indicators comparable to those from a nationally representative … DHS (see Table 3 for an overview of the indicators).’ KIS topics cover family planning, maternal health, child health, HIV/AIDS, and infectious diseases. Their design and content are highly relevant to certain proposed SDG indicators – but they were never fielded. The reason why they were never fielded is that in fact, if a survey was rolled out, the reach was expanded to include many additional indicators. Thus, the lack of adoption of KIS indicates a hunger for data, which is positive. Nevertheless the uptake of shorter surveys could expand if data collection became more regular overall (as has indeed occurred using the remote rapid surveys during the pandemic). The KIS questionnaire and design thus remain a potential resource for this conversation to re-engage.

Table 3. The 20 indicators of KIS.

1.	Total fertility rate	11.	Sanitary practices
2.	Contraceptive prevalence rate	12.	Vitamin A supplementation
3.	Birth spacing	13.	Underweight prevalence
4.	Births to young mothers	14.	Exclusive breastfeeding
5.	High parity births	15.	Drinking water treatment
6.	Skilled delivery assistance	16.	Higher risk sex
7.	Antenatal care	17.	Condom use at higher risk sex
8.	Institutional deliveries	18.	Youth sexual behavior
9.	Childhood immunization coverage	19.	HH availability of insecticide-treated nets
10.	ORT use	20.	Use of insecticide-treated nets

DHS also set up Interim DHS, which ‘focus on the collection of information on key performance monitoring indicators’. Designed to be nationally representative using smaller sample sizes than most DHS surveys, Interim DHS have shorter questionnaires and are conducted between DHS rounds. The Interim DHS surveys have been fielded in Egypt, Guatemala, Jordan, and Rwanda, but again, did not have an enthusiastic take-up. However, like KIS, the survey and sample design are available and can enrich present discussions.

The Core Welfare Indicators Questionnaire (CWIQ) was developed at the World Bank in late 1990s to collect data on the access, usage, and quality of services more frequently than LSMS.²⁰ The core module took roughly 40 min, including anthropometry. At that time, the documents for the CWIQ reported that each household cost $54 in the pilot test reducing to $30 in full survey. Mechanisms to foster data quality included enumerator training and rapid feedback from the questionnaires, which were machine-read, reducing data entry time, and improving accuracy. Timeliness of data and reporting was also stressed, with results being available 6–8 weeks from the end of the fieldwork. Although designed as a stand-alone survey, in many cases, the CWIQ came to be fielded together with a household budget survey or other module, thus losing its quick-ness, but gaining through complementary data. As in the case of KIS, the temporarily expansion of CWIQ is not necessarily a negative finding, given the current infrequency of data collection. An independent evaluation of the CWIQ does not appear to have been conducted, so the status and assessment of this initiative – ranging from the cost to data quality to spread effects such as capacity building – are not yet clear, but could be important to understand for similar initiatives.

These examples – KIS, I-DHS and CWIQ – draw attention to the need to understand fully the ‘demand’ for and ‘inhibitions’ to shortened surveys before embarking. However, they also offer a set of resources on potential questionnaire design and content for consideration in light of the SDGs.

Regional annual surveys with harmonized indicator definitions

The examples above did not address the difficult question of the comparability of survey data across countries. The tradeoff between greater national accuracy and comparability over time (with previous surveys), and greater international comparability, are well-known. What may not be so well-known are the positive examples of annual or biennial surveys that are fielded by NSOs and do include a core of comparable questions.

A noteworthy and rich example for the SDG discussions are the MECOVI surveys in Latin America, which have developed partially harmonized data on 24 Latin American and Caribbean countries for the analysis of poverty and inequality. In many but not all countries, new surveys are fielded annually.²¹ Launched in 1996 and ongoing to this day, MECOVI has increased the capacity of the national statistical systems in undertaking and disseminating analyses from multi-topic household surveys, whilst providing timely and comparable data on key economic, social and living standards indicators. The MECOVI country surveys are not identical, but do cover core variables. In partnership with the World Bank IBRD, and CEPAL, a research center CEDLAS, at University of La Plata, provides support for the harmonization and comparative analysis, including preparation of the SEDLAC database. This database also (like Oxford Poverty and Human Development Initiative's (OPHI) database on the MPI, but focused on this region) includes maps with subnational details of key indicators. The MECOVI programme is longstanding and thoroughly-evaluated, so provides a rich resource for present conversations.

Another relevant example is that of EU-SILC. The European Union Statistics on Income and Living Conditions (EU-SILC) publish annually, timely and comparable cross-sectional and longitudinal multidimensional micro-data on income poverty, social exclusion, and living conditions, now for over 30 countries.²² Anchored in European Statistical System, the EU-SILC project started in 2003 and is ongoing. It may be of interest for the SDG monitoring options because EU-SILC data have been used since 2010 to monitor poverty and social exclusion in the EU towards a target: ‘A headline poverty target on reducing by 20 million in 2020 the number of people under poverty and social exclusion has been defined based on the EU-SILC instrument.’²³

The EU-SILC is replete with interesting lessons. For example, many surveys are only representative at the national level, but some sample sizes are much larger. Certain questions (e.g. levels of education, self-reported health status) may still be difficult to compare across countries (Alkire, Apablaza, and Jung 2014) – an issue that future surveys may address. Also, the use of registry data alongside survey data has been explored in the EU-SILC project, and studies have shown both the potentials and significant difficulties of registry data for poverty monitoring.

One key feature of the EU-SILC process, which could be of tremendous relevance to the SDGs, was the open method of coordination. This method balanced national priorities with progressive harmonization of data and targets.

The open method of coordination, which is designed to help member states progressively to develop their own policies, involves fixing guidelines for the Union, establishing quantitative and qualitative indicators to be applied in each member state, and periodic monitoring. (Atkinson et al. 2002, 1–5)

It may be that for the SDGs, some degree of harmonization across indicators could be advanced in a similar process, at least for some regional or other country groupings. In any case, given the challenges arising from the MDGs’ more top-down measurement agenda, familiarity with alternative processes of data harmonization could be useful.

New technologies: supporting data and transparency

The initiatives reviewed thus far build on tried and tested survey methodologies. In some cases, newer technologies are in use, but by no means in all. New technology has made it possible to extend the reach and speed up the availability of the data, creating a veritable ‘revolution’ indeed. Longer treatments of these technologies with additional examples are collected in a very useful Paris21 Review paper Knowing in Time (Prydz 2014). Here, we focus mainly upon the use of new technologies to facilitate data entry, uploading, analysis, and visualization. However, it should be noted that some important changes to the consent form and survey – for example, retaining the cell phone numbers of respondents for a given set of months – could facilitate monitoring in case of a shock or disaster, by re-contacting respondents with a mini-panel question to ascertain changes in status.

The other bottleneck that these new initiatives are addressing is survey length. For example, a standard consumption/expenditure questionnaire provides a wealth of information on topics ranging from consumption patterns to dietary diversity, to the percentage of income spent on various items, to inequality and distributional issues, and can be analyzed in many ways. Yet if interim annual income and expenditure surveys are used primarily to determine whether or not an individual is income poor, it may be possible to derive this poverty status using shorter modules and imputation, leaving space in surveys to address other core indicators of the SDGs in the years when full consumption/expenditure details are not required.

In terms of promptness and availability, survey programmes have made some important advances, particularly given the more widespread use of Computer-Assisted Personal Interviewing (CAPI) and cloud-based technology. CAPI has a number of features that bolster efficiency and accuracy.²⁴ The immediate transfer of data to central offices permits their immediate analysis. Moreover, such technology is linked with fewer coding errors (as the programme can query errors); enables last-minute updates or corrections to questionnaires; permits dynamic questionnaires (e.g. that enable experiments or asking particular questions based on previous responses); let respondents answer sensitive questions directly without being witnessed and enables more efficient enumerator management.

A signally relevant and rich potential instrument also under development at the World Bank is called the Survey of Welfare via Instant Frequent Tracking (SWIFT). Using a projection method, SWIFT imputes poverty and inequality indicators using models that are calibrated using a country’s previous LSMS or HBS and implemented using core non-monetary indicators. SWIFT has also proposed to include directly the indicators required for a post-2015 MPI (multidimensional poverty index), and questions on subjective well-being (OECD) and consumer sentiment (Eurostat). SWIFT is also taking advantage of CAPI and cloud-based technology to enable the efficient and timely collection, transfer, analysis and release of data.

Other cutting-edge and serious experiments are being undertaken using mobile phones as the medium for a serious of questions on different aspects of well-being (Croke et al. 2012).²⁵ Driven by the same needs as those that motivate the move towards annualized household survey data collection, these forays into ‘high frequency’ survey data are quite certain to strengthen if not transform SDG data collection considerably over the coming decade, but will not replace household surveys in the short and medium term.

Other data collection methods using new technologies explore how to involve the ‘respondents’ more actively in both the data collection and its analysis, so that they – as well as other institutions – can be lead agents of poverty reduction. For example, Paraguay’s Poverty Spotlight are featuring similar technologies – having devised a 20-min visual survey methodology that enables people who are poor to create innovative maps showing the dimensions in which they are poor by using stoplight colors (red, yellow, green), photographs, maps electronic tablets and simple software.

A final note concerns the promptness and availability of the SDG indicators’ publication and construction themselves. Often there is a great silence after data collection has closed before the data are released – a gap the CAPI-cloud technology could shrink. Yet, there is a second delay before the release of official statistics based on those data. Again, some pioneering examples are worth considering. Mexico’s lead institution on poverty measurement and monitoring, CONEVAL, obtains the data from ENIGH (Encuesta Nacional de Ingresos y Gastos de los Hogares). By their own presentations, CONEVAL prepares the official multidimensional poverty statistics (which include income poverty) nationally and by state two weeks after receiving the cleaned data.²⁶ Not only that, but without great delay, the programmes used for calculating poverty are made publically available in STATA, SPSS and R languages, together with a technical note, on the CONEVAL website.²⁷ Thus, academics and technicians can run the programme on the micro-data set (which is also publicly available) to understand, verify the national poverty estimations, and to study and further analyze them. Ecuador’s INEC has a similar online portal with data, algorithms, and poverty analyses.

Conclusion

The move to more frequent reporting of the SDGs is a serious proposition, replete with challenges. There are likely to be shortfalls from the ideal. Yet observing that 60 countries already update data annually, annual updating of a small core set of appropriate poverty-related indicators, and the production of reliable statistics from these data, seems feasible for many countries, and two- to three-year updates of core indicators feasible for nearly all countries – especially since prominent surveys like DHS and MICS are also moving in this direction. The frequent reporting of good quality data with timely data publication and analysis would greatly increase the relevance of measures of poverty to ‘managers’ and policy makers, and these in turn would spark a virtuous cycle. Making micro-data and program files available would increase transparency and increase data analysis by other actors at little cost.

Because of serious and legitimate concerns regarding the realism of increasing data frequency whilst guarding or also increasing the quality of both data and statistics, this section has reviewed a set of positive and negative experiences. We observed that many countries, rather un-noticed, already have annual surveys of some type – and named 60 of them. Most but not all of these are upper middle- and high-income countries. A particularly rich experience appears to be continuous household surveys, which offers the flexibility to update indicators when warranted, decreases issues of seasonality (by fielding over 12 months), and may be more cost effective.

We also observed the challenges faced by international survey initiatives, and the resources already developed for rapid surveys, but these are not cited in the literature calling for more frequent data collection. The hesitant uptake of short surveys points to a hunger for data – which we view to be a real but transitory issue that could subside if data frequency rose. We also reviewed positive examples of nationally implemented yet harmonized indicators which address the need for country ownership and comparability – such as MECOVI and EU-SILC. Both initiatives have interesting processes including data harmonization, financing, the governance roles of international and national bodies, the ongoing role of technical support and a central and standardized data repository. They also are useful to study because they also face ongoing limitations in data quality, sample size, use of registry data, and panel components.

Moving beyond these to consider the timeliness of data, and of non-income indicators, we presented the emerging SWIFT initiative in the World Bank, which fields a short questionnaire to permit the modeling of monetary poverty and direct measurement of multidimensional poverty in a short survey. Aware of the need to communicate poverty results so that they energize and motivate local communities as well as policy makers, we shared the Paraguayan stoplight survey. Finally, in the interests of encouraging transparency of analysis, we shared leading example from countries such as Mexico and Ecuador that post the Stata/SPSS/R files used to compute monetary and multidimensional poverty online, and of generating official national poverty figures rapidly after data release.

This paper skips over many additional vital topics upon which others have written, such as the sequencing of countries moving towards annual surveys, and the important issue of how an increase in data frequency and accuracy can be used to strengthen national statistical systems. Despite these gaps, we hope that the existing conversations, which must address these and other difficult questions, will be facilitated by the information shared here.

Acknowledgements

We are grateful for comments or insightful suggestions from Tony Atkinson, K. Beegal, Gero Carletto, Enrico Giovannini, Attila Hancioglu, Sunita Kishor, Claire Melamed, Jeff Sachs, Emma Samman, Umar Serajuddin and Hiroki Uematsu; and to the Sustainable Development Solutions Network (SDSN) for support. We are also deeply grateful to OPHI colleagues for stellar research assistance and reflective advice, in particular to Felix Stein, Gisela Robles, Usha Kanagaratnam, Joanne Tomkinson Mihika Chatterjee and Christian Oldiges for careful research assistance and to Adriana Conconi, Bouba Housseini, Suman Seth and MPPN colleagues for substantive inputs. For financial support from ESRC/DFID (ES/N01457X/1) we are very grateful.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by Economic and Social Research Council: [Grant Number ES/N01457X/1].

Notes

1 Note that annual GNI data may be subject to issues of accuracy. For example, in 2014, the GNP of Nigeria was re-based. The World Bank’s Nigeria Economic Report (2014) suggest that ‘For the new base year of 2010, the assessed value of GDP increased by 60.7% relative to previous statistics. For 2011, 2012, and 2013, the assessed increases in the level of Nigerian GDP were 68.3%, 76.9%, and 88.9%, respectively.’ I am grateful to K. Beegle for this example.

2 Research addressing monetary and multidimensional poverty is vast. Earlier influential papers include: Atkinson (1987), Foster, Greer, and Thorbecke (1984), Kakwani (1980) and Sen (1976). For more recent work see: Alkire and Santos (2013), Chen and Ravallion (2010), Deaton and Dupriez (2011), Kakwani and Silber (2008) and Ravallion (2017).

3 PovcalNet, corroborated by Umar Serajuddin and Hiroki Uematsu (December 2015).

4 The global MPI (http://www.ophi.org.uk/multidimensional-poverty-index/mpi-2016/) has been estimated and analysed by OPHI, a research centre in the University of Oxford, and published by UNDP’s Human Development Reports since 2010.

5 http://www.globalpolicy.org/home/252-the-millenium-development-goals/52392-csos-appeal-to-high-level-panel.html

6 For example, papers addressing challenges surrounding data availability are: Glewwe (2005), Amarante (2014), Dhongde and Minoiu (2013) and Lemmi et al. (2019).

7 See Schündeln (2018) for discussions on how data quality relates to the frequency of visits.

8 Some use the word poverty to refer to monetary disadvantage, and the word ‘deprivation’ to cover other disadvantages such as malnutrition, low education, ramshackle housing, and so on. We follow the terms used in recent post-2015 agenda documents, which refer to multidimensional poverty, or poverty in all its dimensions.

9 For further discussion of administrate data, public opinion surveys, and big data as resources for poverty data please see Alkire and Samman 2014.

10 We have excluded countries classed as high income by the World Bank, from the main analysis.

11 We have only used the surveys that included the labels: ‘expenditure’, ‘income/income and basic amenities’, ‘income inequality’, ‘budget/budgetary’, ‘household’, ‘consumption’, ‘labour force’, ‘panel surveys’, ‘integrated’, ‘poverty’, ‘priority survey’, ‘welfare’. We excluded all ambiguously or unmarked surveys as well as all surveys that included the labels: ‘agriculture’, ‘census’, ‘consumer finance’, ‘CWIQ’, ‘MICS’, ‘family life’, ‘health’, ‘energy’, ‘panel’, ‘manpower’, ‘housing’, ‘priority’, ‘social’, ‘informal sector’, ‘internally displaced persons’, ‘housing’, ‘service delivery’, ‘social indicators’, ‘social development’, ’socio-economic’, or ‘service delivery’.

12 In 2020, the totals for monetary surveys were 121 countries and 1033 surveys; the figures since 2020 are underestimates as many subsequent surveys have not yet been added.

13 This does not mean we have fully comparable poverty measures for all countries; the surveys generating income and consumption poverty figures are often tailored to national specifications. Still, what we see is a marked rise in data availability.

14 LSMS surveys also measure monetary poverty so are counted as both income and multidimensional surveys. In this period, there were 132 LSMS covering 36 countries, but as they are rarely the only survey in a country they do not affect the total number of countries covered.

15 In a linked paper with Alkire and Samman (2014), we list in Appendix B a set of ‘core indicators of human poverty’ that would come from household survey data, in health and nutrition, education, living standard, work, and violence.

16 For example, in World Development Indicators, a total of 42 countries, both developed and developing, published income poverty data for at least five consecutive years between 2002 and 2012 – but in some cases these published figures are extrapolations, and other countries that have annual data are not included.

17 These are but a sample of surveys as of course other institutions and researchers also have rich data sources. For example, South Africa’s NIDS (National Income Dynamics Survey) is not an official national survey but still provides panel data roughly every two years.

18 Brazil’s PNAD has become a continuous national household sample survey: http://www.ibge.gov.br/english/estatistica/indicadores/trabalhoerendimento/pnad_continua/

19 The KIS website (http://dhsprogram.com/What-We-Do/Survey-Types/KIS.cfm) contains the survey modules.

20 http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/AFRICAEXT/EXTPUBREP/EXTSTATINAFR/0,,contentMDK:21104598~menuPK:3091968~pagePK:64168445~piPK:64168309~theSitePK:824043,00.html ; See also http://www.afdb.org/fileadmin/uploads/afdb/Documents/Publications/African.Statistical.Journal_Vol3_2.Articles_8.ExperiencesApplicationCoreWelfareIndicatorQuestionnaireCWIQ.pdf

21 Details by country are available on: http://www.cedlas.econo.unlp.edu.ar/wp/en/estadisticas/sedlac/estadisticas/

22 EU-SILC Data for 31 countries was available annually for 7 consecutive years between 2006 and 2012. These are: Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland, United Kingdom.

23 http://epp.eurostat.ec.europa.eu/portal/page/portal/microdata/eu_silc

24 See Caeyers, Chalmers, and De Weerdt (2012) for experimental evidence relating to the advantages of CAPI over PAPI (pen-and-paper interviewing).

25 See also their briefing note on http://siteresources.worldbank.org/EXTPREMNET/Resources/EP102.pdf

26 Presentation by CONEVAL, Salamanca, 2013; confirmed by personal conversation with Gonzalo Hernandez Licona, President of CONEVAL.

27 http://www.coneval.gob.mx/Medicion/Paginas/Medici%C3%B3n/Programas-de-Calculo.aspx

Appendix

Appendix A – Multi-Topic Surveys By Country

Table

Country	CWIQ	DHS	ILCS/IS	LSMS	MICS	WHS	Total	First Survey	Last Survey
Afghanistan	0	3	0	0	3	0	6	1997	2019
Albania	0	2	0	7	2	0	11	1996	2018
Algeria	0	0	0	0	5	0	5	1995	2019
Angola	1	3	0	0	2	0	6	1996	2016
Armenia	0	4	11	1	0	0	16	1996	2016
Azerbaijan	0	1	0	1	1	0	3	1995	2006
Bahrain	0	0	0	0	1	0	1	2000	2000
Bangladesh	0	12	0	0	5	1	18	1993	2019
Barbados	0	0	0	0	1	0	1	2012	2012
Belarus	0	0	0	0	3	0	3	2005	2019
Belize	0	0	0	0	3	0	3	2006	2016
Benin	1	5	0	0	1	0	7	1996	2018
Bhutan	0	0	0	0	1	0	1	2010	2010
Bolivia	0	5	0	0	2	0	7	1989	2008
Bosnia-Herzegovina	0	0	0	5	3	1	9	2000	2012
Botswana	1	1	0	0	1	0	3	1988	2010
Brazil	0	3	0	1	0	1	5	1986	2003
Bulgaria	0	0	3	5	0	0	8	1995	2007
Burkina Faso	5	6	0	2	2	1	16	1993	2021
Burundi	1	4	0	0	3	0	8	1987	2017
Cambodia	0	5	0	0	0	0	5	1998	2014
Cameroon	0	5	0	0	3	0	8	1991	2018
Cape Verde	0	1	0	0	0	0	1	2005	2005
Central African Rep.	0	1	0	0	5	0	6	1994	2019
Chad	0	3	0	0	3	1	7	1996	2019
China	0	0	0	1	1	1	3	1995	2003
Colombia	0	7	0	0	0	0	7	1986	2015
Comoros	0	2	0	0	1	1	4	1996	2012
Congo	0	3	0	0	1	0	4	2005	2015
Congo Democratic Re..	0	3	0	0	4	1	8	1995	2018
Costa Rica	0	0	0	0	2	0	2	2011	2018
Cote d'Ivoire	0	4	0	4	4	1	13	1985	2016
Cuba	0	0	0	0	5	0	5	2000	2019
Djibouti	0	0	0	0	1	0	1	2002	2006
Dominican Republic	0	10	0	0	2	1	13	1986	2014
Ecuador	0	1	0	2	0	1	4	1987	2003
Egypt	0	14	0	0	2	0	16	1988	2015
El Salvador	0	1	0	0	1	0	2	1985	2014
Equatorial Guinea	0	1	0	0	1	0	2	2000	2011
Eritrea	0	2	0	0	0	0	2	1995	2002
Eswatini	0	1	0	0	4	0	5	1995	2014
Ethiopia	1	4	0	6	1	1	13	1995	2020
Gabon	2	2	0	0	1	0	5	1996	2017
Gambia	0	1	3	0	5	0	9	1996	2018
Georgia	0	0	22	0	3	1	26	1996	2018
Ghana	5	11	0	5	6	1	28	1987	2019
Grenada	1	0	0	0	0	0	1	2005	2005
Guatemala	0	6	0	1	0	1	8	1987	2015
Guinea	2	5	0	0	2	0	9	1992	2018
Guinea-Bissau	0	0	0	0	6	0	6	1996	2019
Guyana	0	3	0	1	3	0	7	1992	2014
Haiti	0	8	0	0	0	0	8	1994	2018
Honduras	0	2	0	0	0	0	2	2005	2012
India	0	4	0	1	2	1	8	1992	2016
Indonesia	0	12	0	0	2	0	14	1987	2017
Iran	0	0	0	0	2	0	2	1995	1997
Iraq	0	0	0	2	5	0	7	1996	2018
Jamaica	0	0	0	14	2	0	16	1988	2011
Jordan	0	7	0	0	0	0	7	1990	2018
Kazakhstan	0	2	0	1	3	1	7	1995	2015
Kenya	2	11	0	0	2	1	16	1989	2015
Kiribati	0	0	0	0	1	0	1	2018	2019
Korea, Democratic P..	0	0	0	0	4	0	4	1998	2017
Kosovo	0	0	0	1	2	0	3	2000	2020
Kyrgyz Republic	0	2	11	4	4	0	21	1993	2018
Lao PDR	0	1	0	0	5	1	7	1996	2017
Lebanon	0	0	0	0	1	0	1	2000	2002
Lesotho	1	3	0	0	3	0	7	1996	2018
Liberia	3	6	0	0	1	0	10	1986	2016
Libya	0	0	0	0	1	0	1	2003	2007
Madagascar	0	7	0	0	3	0	10	1992	2018
Malawi	7	10	5	8	3	1	34	1992	2021
Malaysia	0	0	0	0	0	1	1	2003	2003
Maldives	0	2	0	0	2	0	4	1995	2017
Mali	1	8	0	3	3	1	16	1987	2020
Mauritania	1	2	0	0	4	1	8	1995	2015
Mauritius	0	0	0	0	0	1	1	2003	2003
Mexico	0	1	0	0	1	1	3	1987	2015
Moldova	0	1	0	0	2	0	3	2000	2012
Mongolia	0	0	0	0	6	0	6	1996	2018
Montenegro	0	0	0	0	3	0	3	2005	2018
Morocco	0	4	0	0	0	1	5	1987	2011
Mozambique	1	6	0	0	2	0	9	1995	2018
Myanmar	0	1	0	0	3	1	5	1995	2016
Namibia	0	5	0	0	0	1	6	1992	2013
Nepal	0	7	0	3	3	1	14	1987	2019
Nicaragua	0	2	0	4	0	0	6	1993	2005
Niger	0	4	0	2	2	0	8	1992	2015
Nigeria	3	8	0	8	5	0	24	1990	2021
North Macedonia	0	0	0	0	4	0	4	1999	2019
Oman	0	0	0	0	2	0	2	1995	2014
Pakistan	0	5	4	1	1	1	12	1990	2019
Palestine	0	0	0	0	5	0	5	1996	2020
Papua New Guinea	0	1	0	0	0	0	1	2016	2018
Paraguay	0	1	0	0	1	1	3	1990	2016
Peru	0	13	0	4	0	0	17	1985	2014
Philippines	0	7	4	0	2	1	14	1993	2017
Qatar	0	0	0	0	1	0	1	2012	2012
Republic of Cabo Ve..	1	0	0	0	0	0	1	2007	2007
Romania	0	0	5	0	0	0	5	1996	2000
Russian Federation	0	0	0	0	0	1	1	2003	2003
Rwanda	1	11	0	0	1	0	13	1992	2017
Saint Lucia	0	0	0	0	1	0	1	2012	2012
Samoa	0	1	0	0	1	0	2	2009	2020
Sao Tome and Principe	0	1	0	0	5	0	6	1996	2019
Senegal	0	22	0	0	2	1	25	1986	2019
Serbia	0	0	0	1	4	0	5	2005	2019
Serbia and Montenegro	0	0	0	2	0	0	2	2002	2003
Sierra Leone	1	5	3	0	5	0	14	1995	2019
Somalia	0	0	0	0	4	0	4	1996	2006
South Africa	0	3	1	1	0	1	6	1993	2016
South Sudan	0	0	0	0	2	0	2	1999	2010
Sri Lanka	0	2	0	0	0	1	3	1987	2007
St. Lucia	2	0	0	0	0	0	2	2004	2012
Sudan	0	1	0	0	4	0	5	1989	2014
Suriname	0	0	0	0	4	0	4	1999	2018
Swaziland	0	0	0	0	0	1	1	2003	2003
Syrian Arab Republic	0	0	0	0	3	0	3	1995	2007
Tajikistan	0	2	0	4	2	0	8	1999	2017
Tanzania	4	14	0	12	1	0	31	1991	2017
Thailand	0	1	0	0	5	0	6	1987	2019
Timor-Leste	0	2	0	2	0	0	4	2001	2016
Togo	3	4	0	0	5	0	12	1988	2017
Tonga	0	0	0	0	1	0	1	2019	2019

Appendix B – reviewed survey sources

This appendix has two parts. Section B.1 lists data portals which can be used to identify national multi-topic household survey data, together with brief descriptions of each portal. Section B.2 lists particular longitudinal multi-topic datasets that include and supplement the examples of EU-SILC and MECOVI covered in this paper.

B.1 data portals

1. Bureau for Research and Economic Analysis of Development (BREAD)

2. Type: Longitudinal

3. Regions: All continents

4. Unit level: Individual/household

BREAD, founded in 2002, is a non-profit organization dedicated to encourage research on development economics. Its website currently locates over 40 types of available household surveys and other data sources about developing countries.

http://www.ipl.econ.duke.edu/bread/

1. CCPR

2. Type: Mostly longitudinal/some cross-sectional

3. Regions: All continents

4. Unit level: Individual/household

Part of UCLA, CCPR’s Survey Database holds over 500 different census datasets and other population surveys from developing countries on demography and reproductive health. The datasets are grouped by regions and type of survey modules, ranging from income over migration and health measurements to time allocation.

http://www.ccpr.ucla.edu/

1. Cross-National Equivalent File (CNEF)

2. Type: Longitudinal

3. Regions: Australia, East Asia, Europe, North America

4. Unit level: Individual

The CNEF contains equivalently defined variables for eight population panel studies: The British Household Panel Study (BHPS, 1991–2008), the Household Income and Labour Dynamics in Australia (HILDA, 2001–2009), the Korea Labour and Income Panel Study (KLIPS, 1998–2008), the Panel Study of Income Dynamics (PSID, 1970–2007) in the United States, the Russia Longitudinal Monitoring Survey (RLMS-HSE, 1995–2010), the Swiss Household Panel (SHP, 1999–2009), the Canadian Survey of Labour and Income Dynamics (SLID, 1993–2009), and the German SocioEconomic Panel (SOEP, 1993–2009).

http://popcenter.uchicago.edu/data/cnef.shtml

1. DataFirst Archive, South Africa

2. Type: Longitudinal/cross-sectional

3. Regions: Africa

4. Unit level: Individual /household

DataFirst is a research unit at the University of Cape Town engaged in promoting the long term preservation and reuse of data from African socioeconomic surveys. Its Data Portal currently provides access to 287 African census-, survey-, and merged meta-data.

http://www.datafirst.uct.ac.za/

1. Eurostat

2. Type: Mostly longitudinal/ some cross-sectional

3. Regions: Europe

4. Unit level: Individual/household/firm

Eurostat is the Statistical Office of the European Communities. Its key role is to provide the European Union with a high-quality statistical information service that enables comparisons between countries and regions. Eurostat’s principal database is the New Cronos – which contains high quality macroeconomic and social statistics data covering not only EU Member States but also many of the central European countries, Japan, the United States and the main economic partners of the EU. The Cronus Database contains monthly, quarterly, bi-annual or annual data from 1960 onwards, depending on the variable and country selected.

http://www.epp.eurostat.ec.europa.eu/

1. INDEPTH Network

2. Type: Mostly longitudinal/ some cross-sectional

3. Regions: Africa, South Asia, East Asia

4. Unit level: Individual

The INDEPTH Network is a global network of 41 health and demographic surveillance system field sites in 20 low- and middle-income countries in Africa, Asia and Oceania, including India. Founded in 1998, its Central Data Catalogue currently holds 19 surveys.

http://www.indepth-ishare.org/

1. Integrated Public Use Microdata Series International (IPUMS International)

2. Type: Longitudinal

3. Regions: All continents

4. Unit level: Individual

IPUMS International is a collaboration of the Minnesota Population Centre, National Statistical Offices, and international data archives aiming to distribute harmonized population census micro-data. The database currently features censuses from 74 countries conducted from 1960 to the present, and describes approximately 545 million recorded persons. The data series includes information on a broad range of population characteristics, including fertility, nuptiality, life-course transitions, migration, labor-force participation, occupational structure, education, ethnicity, and household composition. The information available in each sample varies according to the questions asked in that year and by differences in post-enumeration processing.

http://www.international.ipums.org/international/

1. International Food Policy Research Institute (IFPRI)

2. Type: Longitudinal

3. Regions: Africa, Asia, Latin America

4. Unit level: Household/community

IFPRI currently shares 99 of its datasets, which feature both household/community level surveys and social accounting matrixes. The household and community surveys include several surveys of household characteristics, consumption and health as well as agricultural information and food security information, while the social accounting matrices are an economic framework study with a focus on agriculture. Some studies include geospatial data. IFPRI also publishes implementation, monitoring and implementation data, for instance on cash transfer implementation.

http://www.ifpri.org/

1. Inter-University Consortium for Political and Social Research (ICPSR)

2. Type: Longitudinal/cross-sectional

3. Regions: All continents

4. Unit level: Individual/household

The ICPSR is an international consortium of academic organizations and research institutions established in 1962. It maintains and provides access to a vast archive of social science data, featuring over 8,000 discrete studies/surveys with more than 60,000 datasets. Apart from offering a topic- and regional-specific search, ICPSR hosts 16 discipline-related thematic collections in education, aging, criminal justice, demographic data, health and mental health, instructional data, race and ethnicity, and terrorism.

http://www.icpsr.umich.edu/

1. International Household Survey Network (IHSN)

2. Type: Longitudinal/cross-sectional

3. Regions: All continents

4. Unit level: Individual/household

The IHSN Central Survey catalogue provides a searchable list of surveys and censuses conducted in low- and middle-income countries. This catalogue is maintained in collaboration with the World Bank and a large number of national and international agencies. Currently, it features 4221survey entries from 239 countries, dating from 1890 to 2014. The catalogue offers metadata including, when available, the survey questionnaire, manuals and report, and list of related citations. It does not provide access to micro-data, but when available, provides a link to external catalogs where the data can be obtained.

http://www.ihsn.org/home/

1. Programme for the Improvement of Surveys and the Measurement of Living Conditions in Latin America and the Caribbean/ Mejoramiento de las Encuestas de Hogares y la Medición de Condiciones de Vida (MECOVI)

2. Type: Longitudinal/cross-sectional

3. Regions: Latin America

4. Unit level: Household

MECOVI was launched in 1996 and aims to generate both country-specific and region-wide information about living conditions. The program is executed by the World Bank, the Inter-American Development Bank and the United Nations Economic Commission for Latin America and the Caribbean, as well as specialized institutions or agencies in participating countries. Apart from its work around national statistical capacity building, MECOVI has created a Regional Poverty Data Bank that contains an inventory of more than 400 household survey data sets from 23 countries in the LAC region. The data sets are accessible to World Bank users or via the respective National Statistical Offices.

http://www.cepal.org/deype/mecovi/

1. Rural Income Generating Activities (RIGA) Database

2. Type: Longitudinal/cross-sectional

3. Regions: Africa, Asia, Eastern Europe, Latin America

4. Unit level: Household

RIGA is a collaborative effort of FAO, the World Bank and American University in Washington, DC, to promote the understanding of roles, relationships and synergies between on-farm and off-farm income generating activities for rural households. Building on existing household living standards surveys, the database contains cross-country comparable indicators of household-level income for 35 surveys representing 19 countries, with surveys conducted between 1992 and 2009.

http://www.fao.org/economic/riga/riga-database/en/

1. UCLA Social Science Data Archive (SSDA)

2. Type: Longitudinal/cross-sectional

3. Regions: Mostly US, but all other continents as well

4. Unit level: Individual/household

The SSDA, founded in 1964, is maintained so as to provide a foundation for social science research as well as instructional support. Its current list of data sets features around 3000 items, many of them older surveys.

http://www.dataarchives.ss.ucla.edu/

1. UK Data Service

2. Type: Longitudinal/cross-sectional

3. Regions: All continents

4. Unit level: Individual/household

The UK Data Service, funded by the Economic and Social Research Council (ESRC), provides access to secondary social and economic data including large-scale government surveys, international macro-data, business micro-data and census data from 1971 to 2011. It currently features over 6,000 datasets that are arranged by survey type (UK surveys, cross-national surveys, longitudinal studies, census data, international macro-data, business micro-data, qualitative methods) as well as core themes (labor market, housing and the local environment, crime and social control, health and health behavior). The UK Data Service was established in 2012 and previously existing data archives such as the Economic and Social Data Service (ESDS) have been moved to it in order to create a single portal.

http://www.ukdataservice.ac.uk

B.2 data sets

Table 2.1 Reviewed Data Sets.

Name	Description	Reference Portal(s)
Region: Africa
Ethiopia Rural Household Survey	Panel data set by the Centre for the Study of African Economies at Oxford University covering households in a number of villages in rural Ethiopia. Data collection took place in the period from 1989 until 2009 in altogether 7 waves, surveying about 1470 households.	IFPRI
Ghana and Tanzania Urban Household Panel Surveys	Labor market panel survey of urban sectors in Ghana and Tanzania, conducted by the Centre for the Study of African Economies at Oxford University in collaboration with the Ghana Statistical Office and the Tanzania National Bureau of Statistics. From 2004 until 2006, three waves of the survey have been completed. The survey collects information on incomes, education and labor market experience, household characteristics and various other modules for labor force participants (ages 15–60) in urban areas.	CSAE
Kenya and Malawi Social Networks Projects	Since 1998, the Malawi Longitudinal Study of Families and Health and the Kenya Diffusion and Ideational Change Project collect longitudinal socio-demographic data on social interactions, changing demographic attitudes and health conditions.	BREAD
SALDRU Langeberg Survey	Integrated household survey undertaken in 1999 in the South African Langeberg health district of the Western Cape. Information on adult and child health was collected from a 294 stratified household sample.	BREAD
South African National Income Dynamics Study (NIDS)	Nationally representative panel study that examines income, consumption and expenditure of households over time in South. Africa. The baseline survey was conducted in 2008 and the first follow-up was conducted in 2010. Three waves have been implemented so far. In addition to income and expenditure dynamics, study themes include the determinants of changes in poverty and well-being, household composition and structure, fertility and mortality, migrant strategies, labor market participation and economic activity, human capital formation, health, education, vulnerability and social capital.	BREAD
Region: Asia
Cebu Longitudinal Health and Nutrition Surveys (CHLNS)	On-going study of a cohort of Filipino women who gave birth between May 1, 1983 and April 30, 1984 and have been re-interviewed in five waves since then. In 1994 a new cohort was added to the study. Research is focused on the long-term effects of prenatal and early childhood nutrition and health on later adult outcomes including education, work, and chronic disease risk factors.	BREAD
China Health and Nutrition Survey	On-going longitudinal study first conducted in 1989 in 8 provinces in China. It provides information on health and nutrition of adults and children, as well as community level data.	BREAD
China Health and Retirement Longitudinal Study (CHARLS)	On-going longitudinal survey patterned after the US Health and Retirement Study. Two nationally representative waves of people 45 and over have been conducted in 2011 and 2013.	BREAD
India Agriculture and Climate Data Set	Database providing district level data on agriculture and climate in India from 1957/58 through 1986/87. The dataset includes information on agricultural labor, wages and factory earnings, rural population and literacy proportion, soil quality, production, farm harvest prices and agricultural inputs.	BREAD
India Human Development Survey (IHDS)	Nationally representative multi-topic longitudinal survey of over 41,000 households in India. The baseline was conducted in 2004-5.	BREAD
Indian States Data (EOPP)	Indian state-level micro- and macro-data compiled by the Economic Organization and Public Policy Programme at the LSE. Topics covered include land reform, media and political agency, quality of life, and economic reforms.	BREAD
Indonesia Family Life Survey (IFLS)	On-going longitudinal survey with four waves from 1993/94 until 2007 Conducted by RAND. The data collected at the individual, household and community level in 13 of 27 provinces is representative of about 83% of the Indonesian population. The surveys include household consumption, assets, health measures, and retrospective histories on, among others, employment, marriage, fertility and migration.	BREAD
Learning and Education Achievement in Punjab Schools (LEAPS)	Panel project by researchers at Harvard University, Pomona College, and the World Bank that tracks changes in educational universe at the primary level in 112 villages in Pakistan. Children, households, schools and teachers are followed over four waves from 2001 to 2005.	BREAD
Malaysian Family Life Surveys (MFLS)	Longitudinal survey with two waves in 1976/7 and 1988. Conducted by RAND. Surveys include detailed current and retrospective information on family structure, fertility, economic status, education/training, transfers and migration. Each survey also collected community-level data.	BREAD
Matlab Health and Social Survey, Bangladesh (MHSS)	Conducted in 1996 by RAND and covering the same area as the Matlab Demographic Surveillance System. The survey examined the effect of socioeconomic and behavioral factors on adult and elderly health status and health care utilization as well as the linkages between well-being, social network characteristics and resource flows.	BREAD
Nang Rong (Thailand) Projects	The Nang Rong Projects was started in 1984 with a census of households in 51 villages, resurveyed in two waves in 1988 and 1994. Data on life course choices, fertility, contraceptive behavior and migration processes is integrated with geographic and environmental information.	BREAD
National Sample Survey Organization (NSSO)	The Indian National Sample Survey Organization conducts multi-subject integrated sample surveys, with both central government and state samples. Information on social, economic, demographic, industrial and agricultural activity is provided within 10-year subject timeframes.	BREAD
Rural Economic and Demographic Survey (REDS)	Rural household and village survey carried out in five waves from 1969 to 1999 by the Indian National Council of Applied Economic Research. Some of the respondents have been interviewed in several rounds yielding a panel spanning 30 years.	BREAD
Survey on the Status of Women and Fertility (SWAF)	Comparative 1993/1994 study of the status of women and their husbands in conjunction with fertility choices in Malaysia, India, Pakistan, the Philippines and Thailand.	BREAD
The Townsend Thai Project	On-going longitudinal study comprising annual and monthly panels. The baseline survey was conducted in 1997 in villages in four provinces and has been expanded to add urban areas and other provinces.	BREAD
Vietnam Life History Survey	The 1991 survey collects data from about 100 households in two urban and two rural areas in Vietnam.	BREAD
Vietnam Longitudinal Survey	Longitudinal survey with three rounds between 1995 and 1988. The survey collected demographic information from all adult respondents in over 1,800 households in three provinces.	BREAD
Region: Europe
Adult Education Survey (AES)	The AES household survey forms part of a wider set of EU statistics on lifelong learning. It covers participation in education and training activities (formal, non-formal and informal learning) of persons aged between 25 and 64. Two survey waves (2007 AES, 2011 AES) have been carried out so far in 29 countries with EU membership, EU candidate or EFTA status. The AES is planned to be conducted every 5 years, with the next wave in 2016.	Eurostat
European Community Household Panel (ECHP)	The ECHP is a transnational panel survey in which a sample of roughly 60,500 nationally represented households (equating to some 130,000 persons aged 16 years and over in 15 countries) were interviewed on an annual basis from 1994 to 2001 (8 waves). The survey covers a wide range of topics concerning living conditions. They include detailed income information, financial situation in a wider sense, working life, housing situation, social relations, health and biographical information. As from 2003/2004, the EU-SILC survey covers most of the above-mentioned topics.	Eurostat, UK Data Service
European Social Survey (ESS)	The ESS is a biennial multi-country survey covering over 30 nations. The first round was fielded in 2002/2003; the sixth in 2012.The ESS provides data on the interaction between Europe’s changing institutions and the behavior, beliefs and attitudes of European citizens. Amongst other variables this includes data on social exclusion, well-being, health, security, demographics and socioeconomics.	Eurostat, UK Data Service
European Structure of Earnings Survey (SES)	This survey provides harmonized data on earnings in EU member states, countries of the European Free Trade Association as well as EU candidate countries. It was conducted in 2002 and 2006 in 29 countries. It is not a household survey but focuses on enterprises with at least 10 employees. The 4-yearly SES micro-data sets are available for reference years 2002, 2006 and 2010.	Eurostat
European Union Labour Force Survey (EU – LFS)	The EU-LFS is a cross-sectional and longitudinal household sample survey. It provides data on labor participation in the 28 Member States of the European Union, 2 candidate countries and 3 countries of the European Free Trade Association. Since 1983, a revised annual survey with quarterly employment data is conducted. In 2011, the quarterly LFS sample size across the EU was about 1.5 millions of individuals. The EU-LFS covers all industries and occupations.	Eurostat, UK Data Service
European Union Statistics on Income and Living Conditions (EU-SILC)	EU-SILC collects cross-sectional and longitudinal micro-data on income, poverty, social exclusion and living conditions. It was first carried out in 2003 and provides data for most EU member states as well as Turkey. Cross sectional data is released every year in March while longitudinal data is provided every August as from 2010. Social exclusion and housing condition information is collected mainly at household level while labor, education and health information is obtained for persons aged 16 and over. The core of the instrument, income at very detailed component level, is mainly collected at personal level.	Eurostat, UK Data Service
Russia Longitudinal Monitoring Survey (RLMS)	On-going panel survey of Russian households that began in 1992 and collects data on individuals’ health status and dietary intake as well as household-level expenditures and service utilization. In 2013, 22 rounds had been conducted.	BREAD
Region: Latin America and the Caribbean
Central American Population Project	Collects fertility and health surveys carried out in Central America. Data from Belize, Guatemala, El Salvador, Honduras, Nicaragua, Costa Rica and Panama are included in the collection.	BREAD
Guatemalan Survey of Family Health (EGSF)	Single cross section survey conducted in 1995 in rural communities in 4 of Guatemala’s 22 departments. The survey examined the way in which rural Guatemalans cope with childhood illness and pregnancy, and the role of ethnicity, poverty, social support, and health beliefs.	BREAD
Mexican and Latin American Migration Project (MPP, LAMP)	On-going longitudinal study of Mexican Migration to the US. Its annual survey waves cover Mexican households since 1982, with special sub-samples of Mexicans living in Chicago. In extension to the MPP, the LAMP has collected data in Puerto Rico, the Dominican Republic, Nicaragua, Costa Rica and Peru since 1988.	BREAD
Mexican Family Life Survey (MxFLS)	On-going nationally representative longitudinal survey of individuals, households, families and communities. Conducted by RAND. The first wave was conducted in 2002, with two follow-ups so far. In addition to consumption, income, wealth, employment, marriage and fertility, the survey contains a module on crime and victimization as well migration histories.	BREAD
Mexican Health and Aging Study (MHAS)	Prospective longitudinal survey of older adults (born before 1951) and their spouses. 10,000 adults and 5,000 spouses were interviewed in the first 2001 wave, with a follow-up completed in 2003. A fourth round of the longitudinal study is planned for 2015.	BREAD
SABE (Salud Bienestar Y Envejeveimiento en America Latina y El Caribe)	Series of comparable cross-national surveys on health and aging organized as a cooperative venture among researchers in Argentina, Barbados, Brazil, Chile, Cuba, Mexico and Uruguay. Its goal is to describe health, cognitive achievement and access to health care among people age 60 and older with a special focus on people over 80 years old.	BREAD
Tsimane Amazonian Panel Study (TAPS)	TAPS is an annual panel data set covering the period 2002 through 2006 that follows a native Amazonian horticultural and foraging society. The study has been tracking about 1,500 native Amazonians in about 250 households of 13 villages along the Maniqui River in Bolivia.	BREAD
Region: Global/Multi-Regional
Core Welfare Indicator Questionnaire (CWIQs)	The World Bank developed the CWIQ survey series in the 1990s as an inexpensive tool to collect standardized information on poverty, including access and satisfaction with social services and social welfare indicators. The surveys contain information related to housing conditions, water and sanitation, education, health care use and access, income and assets.	IHSN
Demographic and Health Surveys (DHS)	DHS is collecting national sample surveys of population and maternal and child health. It includes a range of data collection options. Individual and household level data has been recorded in many developing countries since the 1980s. Data have been collected in four waves: DHS-I (1986-90), DHS-II (1991-1992), DHS-III (1993-1997), Measure (1998-present).	BREAD, STICERT
Living Standards Measurement Studies (LSMS)	Since 1980, the World Bank has been collecting multi-purpose household survey data in 39 countries under the Living Standards Measurement Study umbrella. The LSMS-Integrated Surveys on Agriculture Project (LSMS-ISA) conducts surveys and research on the links between agriculture and poverty reduction.	BREAD, STICERT
Multiple Indicator Cluster Survey (MICS)	International household survey initiative by UNICEF producing internationally comparable estimates of a range of indicators in the MDG target areas of health, education, child protection and HIV/AIDS. The first MICS round was carried out in 1995 in more than 60 countries, and has been followed by four waves so far, with the fifth wave still running in 2014.	IHSN
Young Lives: An International Study of Childhood Poverty	The Young Lives study, which began in 2002, is an innovative long-term project investigating the changing nature of childhood poverty in Ethiopia, India, Peru and Vietnam. It is following 12,000 children in these countries over 15 years. It is conducted by the Young Lives team based at the University of Oxford.	UK Data Service