How to Target Households in Adaptive Social Protection Systems? Evidence from Humanitarian and Development Approaches in Niger

Abstract

The methods used to identify the beneficiaries of programmes aiming to address persistent poverty and shocks are subject to frequent policy debates. Relying on panel data from Niger, this report simulates the performance of various targeting methods that are widely used by development and humanitarian actors. The methods include proxy-means testing (PMT), household economy analysis (HEA), geographical targeting, and combined methods. Results show that PMT performs more effectively in identifying persistently poor households, while HEA shows superior performance in identifying transiently food insecure households. Geographical targeting is particularly efficient in responding to food crises, which tend to be largely covariate. Combinations of geographical, PMT, and HEA approaches may be used as part of an efficient and scalable adaptive social protection system. Results motivate the consolidation of data across programmes, which can support the application of alternative targeting methods tailored to programme-specific objectives.

1. Introduction

Adaptive social protection (ASP) systems are increasingly popular instruments for reducing poverty and mitigating the impacts of shocks. ASP entails a twofold approach. First, it reduces poverty and builds resilience before shocks occur through predictable transfers, building community assets, and other programmes that promote livelihoods. Second, it quickly scales up interventions in response to shocks (World Bank, 2018). In practice, development and humanitarian actors have often addressed these dual goals with little coordination, resulting in an inefficient use of the resources allocated to poor countries and missed opportunities to produce larger intervention impacts. Greater integration across interventions presents a key opportunity to establish effective ASP systems (Davies et al., 2013). This integration should embody a range of elements, including funding and institutional aspects, the nature of benefits, and targeting methods.

Development and humanitarian interventions often rely on different approaches to targeting. This may occur for various reasons. First, there are inherent differences in objectives in addressing persistent versus transient deprivations, which may also relate to specific dimensions of well-being in the face of a crises (such as in food, water, or shelter). Second, the emergency nature of humanitarian interventions may require quick beneficiary identification processes. Finally, access, security, and limited staff capacity in the context of an emergency may make some targeting choices unfeasible or less appropriate (Maxwell, Young, Jaspars, Frize, & Burns, 2011). The understanding is limited on how development and humanitarian targeting approaches compare, their potential synergies, and how they could be integrated as part of an ASP system.

Relying on panel data from Niger, this report analyses the performance of different targeting methods that are widely used by development and humanitarian actors and discusses how they could be applied as part of an ASP system. The methods include proxy-means testing (PMT) and household economy analysis (HEA), as used in rural Niger by a development programme and by a humanitarian programme addressing food crises, respectively. The study also compares PMT and HEA with geographical and combined targeting methods.

The study contributes to the literature on the targeting of social protection programmes in different ways. First, while HEA is widely used in the Sahel to identify beneficiaries of humanitarian programmes, no evidence exists in the academic literature on its performance. Second, by relying on panel data, the study also contributes to the limited literature on how targeting methods can help identify households with a dynamic welfare status (Del Ninno & Mills, 2015). Welfare can be largely volatile, and recognising these dynamics is key to improving targeting effectiveness (United Nations Children’s Fund [UNICEF], 2017). Finally, the study compares several of the most widely used methods and contributes to the limited understanding on how these can be applied as part of ASP systems.

Niger, which suffers from high poverty rates and recurrent shocks, offers relevant lessons for other countries trying to build ASP systems. With a gross domestic product per capita of US$895, Niger was the sixth poorest country in the world in 2014. Humanitarian needs continue to be significant because of persistent food insecurity, which has been exacerbated by climate variability, price fluctuations, and waves of political instability. Even in good agricultural years, four million to five million people experience food shortages (European Commission, European Civil Protection and Humanitarian Aid Operations [ECHO], 2018).

Moreover, while the needs are large, budgets are small and can cover only a small portion of the population (Beegle, Coudouel, & Monsalve, 2018). Multiple actors also operate, and there is a range of programmes using different approaches to select beneficiaries. A better understanding and integration of appropriate targeting methods are considered core elements in improving coordination and policy effectiveness.

Results show that PMT and HEA target two largely different types of population. PMT performs more effectively in targeting persistently poor households, while HEA performs better in targeting transient food insecure households. Specifically, PMT presents, by 29 percentage points, fewer inclusion errors than HEA based on persistent poverty rates, but, by 18 percentage points, more inclusion errors than HEA based on transient food insecurity. Differences in the selection of households are driven by both the choice and the weights assigned to variables used to identify beneficiaries.

If the objective is to identify transient food insecure households, the simulated geographical approach performs at least as good as or substantially more accurately than any other approach tested in the research discussed in this study. In contrast, if the aim is to identify persistently poor households, the performance of the simulated geographical approach is moderately inferior relative to the PTM approach or to a combined approach (presenting, respectively, fewer inclusion errors by 11 and 14 percentage points).

The next section describes the various targeting methods analysed in this study. The following section describes the data and welfare benchmarks that are used to evaluate targeting efficiency. The performance of PMT and HEA is presented in the subsequent section. The succeeding section compares PMT and HEA with various geographical and combined targeting approaches. The penultimate section discusses costs, and the final section concludes.

2. Description of the targeting methods

2.1. Proxy-means testing

The PMT method relies on a limited set of household characteristics that are applied in a formula approximating household income or consumption (Grosh & Baker, 1995). In contexts in which the means-testing of benefits is not an administratively feasible option, as in most low-income settings, PMT provides the advantage of relying on information that can be measured relatively quickly and that cannot be easily manipulated. Based on various studies on sub-Saharan Africa, Del Ninno and Mills (2015) suggest that, while PMT can effectively identify households suffering from persistent poverty, its efficacy in identifying households in the context of a crisis may be limited, given its substantial reliance on long-term household characteristics.

While much research on PMT exists, papers assessing PMT performance relative to other methods are less common. Papers comparing the performance of PMT and community-based targeting (CBT) methods find that PMT outperforms CBT based on consumption per capita, even if, in some cases, the performance gap is not large (Alatas, Banerjee, Hanna, Olken, & Tobias, 2012; Karlan & Thuysbaert, 2016; Premand & Schnitzer, 2018; Stoeffler, Mills, & Del Ninno, 2016). Nonetheless, evidence has suggested that CBT seems to be focused on factors other than consumption, such as livestock and land, human and physical capital asset holding, and household earning capacity (Alatas et al., 2012; Karlan & Thuysbaert, 2016; Stoeffler et al., 2016).

The PMT evaluated in this study was implemented by the government as part of the first phase of the World Bank–funded Social Safety Nets Project, which sought to target 40,000 persistently poor rural households in five regions. The project provided eligible households monthly cash transfers amounting to CFAF 10,000 and accompanying measures for 24 months. To identify the population of interest, geographical and household targeting approaches were used. For the geographical targeting approach, various poverty indicators, together with population data, were used to select and allocate beneficiary shares to regions, departments, and communes. Next, all villages were considered eligible for the cash transfer programme, and, given the lack of poverty information on the villages and to ensure transparency, beneficiary villages were selected through public lotteries.

The household targeting approach involved a short census among all beneficiary villages. The census allowed the application of the PMT formula to identify 30 per cent of the poorest households in the selected villages within each commune. This was followed by a community validation process. The PMT formula was developed based on the third national household budget and consumption survey (Troisième Enquête Nationale sur le Budget et la Consommation des Ménages) collected in 2007. The formula’s objective was to proxy for per capita consumption. It relied on a set of variables, including household demographics, livestock, land, and other productive and non-productive assets.¹

2.2. Household economy analysis

HEA is a livelihoods-based framework developed to improve the ability of humanitarian agencies to anticipate and respond to food crises (Holzmann, 2008). The approach produces a nationwide HEA baseline based on a qualitative assessment.² Within each livelihood zone (defined as part of the HEA baseline), the assessment identifies three or four wealth categories according to clearly defined and measurable household characteristics. Since its inception, the HEA baseline has been developed and adapted for various purposes, including household targeting, which is the focus of this study.

To target beneficiaries, HEA has traditionally relied on a CBT exercise that categorises households into three or four wealth groups based on specified criteria that are agreed by the community and largely guided by the HEA baseline. Normally, the households categorised in the poorest group are selected as beneficiaries. Recognising that HEA is a tool used in different ways and with different objectives, HEA refers hereafter in this report to the HEA approach applied in targeting households.

The HEA method evaluated in the study was developed and used by Alliance ECHO in Niger to provide temporary cash transfers over three to four months that were aimed at supporting food insecure households during the lean season.³ The approach is applied only after geographical areas are selected based on the national famine early warning system, which relies on the Cadre Harmonisé (see below).

The approach is implemented following two parallel steps involving a CBT exercise and the application of a formula and then a final triangulation step. During triangulation, any discrepancies in the selection results emerging from the parallel steps are addressed by nongovernmental organisations, together with community members.

During the CBT exercise, two targeting committees of about eight members each are formed. (Normally, one is composed of women, and the other of men.) Using an exhaustive list of all households in the communities, each committee independently classifies each household into one of four wealth groups, following predetermined criteria, which are discussed before the ranking exercise starts and are based on the criteria determined for the HEA baseline.

For the application of the formula, a short survey is undertaken among every household in the community. The formula was developed qualitatively in 2014 based on historic targeting data of HEA users, who, at the time, would rely only on the CBT exercise to identify beneficiaries. The objective of the formula was to possess more systematic and objective criteria in the selection of households.⁴ The HEA formula partly relied on the same information used by the PMT formula, notably household demographics, livestock, and land indicators. Other information was used only by the HEA formula (the duration of food coverage based on own agricultural production and monthly household revenue) or only by the PMT formula (other productive assets and non-productive assets). Differences in the choices of variables and the weights of variables seem to drive differences in the beneficiary selection results (Table 1).

Table 1. Explaining differences between households selected by PMT and by HEA

Variable	Coefficient
Household size	0.0355***
	(0.00538)
Woman-headed household	0.100*
	(0.0524)
Polygamous family	0.0805*
	(0.0443)
Has cart	−0.0306
	(0.0424)
Has hoe	−0.157**
	(0.0724)
Number of small ruminants	−0.0269***
	(0.00464)
Number of large ruminants	−0.0786***
	(0.0169)
Cultivated land	0.000246
	(0.00105)
Non-Productive asset index	−0.0539***
	(0.0144)
Average monthly revenues in cash	−1.28e−06**
	(5.34e−07)
Food coverage based on own production (months)	−0.0156**
	(0.00675)
Number of sectors in which the household is engaged	−0.0397*
	(0.0209)
Share of income from remittances	−0.000423
	(0.0331)
Observations	864
Mean of Y	0.434

Notes: Dependent variable = 1 if PMT and HEA resulted in different beneficiary selection outcomes; 0 otherwise. Robust standard errors in parentheses. A probit model was used, and marginal effects are reported. p-values derived from tests of equality: * = 10 per cent, ** = 5 per cent, *** = 1 per cent.

2.3. Geographical targeting

Social protection programmes typically rely on geographical targeting either in isolation or combined with other household targeting approaches. While development actors have usually depended on poverty maps or other econometric techniques to select areas, humanitarian actors have often favoured early warning systems.

Poverty maps combine household survey information on consumption or income (available on relatively large geographical areas) with other data sources (available at a smaller geographical scale) to estimate poverty rates in small geographical areas. Based on poverty maps drawn up on in three countries, Elbers, Fujii, Lanjouw, Özler, and Yin (2007) conclude that geographical targeting on small geographical areas can result in large efficiency gains. Nonetheless, they suggest that the targeting performance is far from perfect and that combining geographical targeting and household targeting methods may be important in improving efficiency. While poverty maps have traditionally relied on census data, which are not always available in low-income settings, novel approaches focused on machine learning techniques and satellite data imagery that are publicly available have been shown to work well (Jean et al., 2016).

Early warning systems combine selected data sources to identify hazards in a timely manner. Little is known about how they can identify populations in need (Enenkel et al., 2015). Famine early warning systems in the Sahel and West Africa have usually relied on the Cadre Harmonisé, a regional framework for consensual analysis of food insecurity situations.⁵ The Cadre Harmonisé relies on combination of data, including household assessments, crop production and balance sheets, climate data, anthropometric data, and prices. However, concerns about its reliability and objectivity have been expressed, and its heavy dependence on frequent data collection makes the process costly and lengthy. Whether and how readily available satellite imagery, combined with machine learning techniques (such as the method used in Jean et al., 2016), could be used to produce results that are less reliant on data collection remain open questions.

3. Data and welfare benchmarks

3.1. Data

This report is founded on data of the 2011 Niger National Survey on Household Living Conditions and Agriculture (ECVM/A). The 2011 ECVM/A was a multitopic household survey conducted to quantify poverty and living conditions in Niger during the lean and harvest seasons, that is, June–August 2011 and October–December 2011.⁶ Household composition, consumption per capita, and the food consumption score (FCS) were measured during both rounds, while all other welfare indicators used in this report were measured once.

The survey collected data from a total of 3,968 households. The sample is representative at the national level as well as in Niamey, in urban areas (excluding Niamey), and in rural areas. Within rural areas, the sample is also representative of three livelihood zones: agricultural, agropastoral, and pastoral. The sample in this study is restricted to the agropastoral zones only (N = 867). This is because the HEA method evaluated in the study was developed exclusively for agropastoral zones. Caution is therefore needed in extrapolating the results of the study to other parts of Niger, where poverty and food insecurity rates and determinants may be different. Agropastoral areas are characterised by high interannual rainfall variability and represent a high priority zone in monitoring food insecurity. Humanitarian and development interventions are widespread in these areas. The study makes use of survey household sampling weights throughout to maintain representativeness.

3.2. Welfare benchmarks

In designing a targeting system, it is important, first, to define clearly who the intended beneficiaries to be reached are and, then, how to measure the performance in reaching them. While targeting households with low levels of consumption is often assumed to be reasonably effective in reaching households facing other deprivations, evidence shows that this is not always the case (Barrett, 2010; Brown, Ravallion, & van de Walle, 2017; UNICEF, 2017).

Defining and measuring welfare benchmarks are not easy tasks. Different views exist on what being poor or vulnerable to shocks means and how to measure these outcomes. While the literature on targeting methods is vast, much of it has focused on identifying the poor (measured through a snapshot of income or consumption), but leaving important gaps in how to identify other groups, especially those affected by crises (Alatas et al., 2012; Brown et al., 2017; Del Ninno & Mills, 2015; Devereux et al., 2017; Karlan & Thuysbaert, 2016; Stoeffler et al., 2016).

Who the intended beneficiaries of interventions are can vary within and across humanitarian and development interventions. Humanitarian actors are usually concerned with supporting households experiencing transient deprivations in the face of a crisis. These deprivations may concern different aspects of welfare, such as food insecurity; however, these aspects are often loosely defined (Barrett, 2010). Meanwhile, development actors are most often focused on supporting households that experience persistent poverty, normally measured through consumption.

To proxy the objectives of various programmes and measure targeting performance, this study relies on two main welfare benchmarks: (a) the average consumption per capita between the lean and harvest seasons as a measure of persistent poverty and (b) the FCS during the lean season as a measure of transitory food insecurity.⁷ The FCS is among the most widely used indicators among humanitarian actors. It is designed to reflect both the quantity and the quality of the food consumed. It is based on a set of questions on consumption frequency across various food groups during the seven days prior to the survey (World Food Programme, Vulnerability Analysis and Mapping Branch [WFP VAM], 2008).⁸

Figure 1 illustrates the prevalence of poverty and food insecurity during the lean and harvest seasons in agropastoral zones in Niger.⁹ It highlights the relatively persistent and transient nature of poverty and food insecurity, respectively. During the lean season, 37 per cent and 21 per cent of households experienced persistent and transient poverty, respectively. In contrast, 8 per cent and 25 per cent of households experienced persistent and transient food insecurity, respectively.

Figure 1. Poverty and food insecurity during the lean and harvest seasons.

Notes: Households experiencing persistent poverty or food insecurity are below the poverty line or food insecure threshold in both 2011 survey rounds, that is, during the lean and harvest seasons. Households experiencing transient poverty or food insecurity are below the poverty line or food insecure threshold during one season only. Corresponding tables can be found in the Supplementary Appendix.

4. The performance of PMT and HEA

4.1. Simulating the selection of PMT and HEA beneficiaries

This report analyses only the performance of the PMT and HEA formulas in selecting households. It does not consider other steps used in the process of selecting beneficiaries. In particular, in both cases, a community-based approach is used to validate or triangulate beneficiary lists obtained from the formulas. However, in practice, none of the community-based approaches significantly change the beneficiary lists obtained through the formulas. In the case of PMT, the final beneficiary list obtained from the community validation process almost entirely overlaps (around 99 per cent) with the list obtained from the formula.¹⁰ Similarly, in the case of HEA, the community-based results have an overlap of 90 per cent with the formula results (Bourahla, Evrard-Diakite, Malam, & Boulinaud, 2014). Also, PMT and HEA are both implemented after a geographical targeting exercise aimed at identifying poor and food insecure areas, respectively. While the study simulates geographical approaches in combination with PMT and HEA, it is not able to address how the geographical approaches applied in practice may affect the targeting efficiency of PMT and HEA.

To evaluate targeting performance, the selection procedure is simulated based on the PMT and HEA formulas. Using the ECVM/A data, the simulation provides both a PMT score and an HEA score for each household, which represent the predicted level of welfare of the household. The scores are used to rank all households from least well off (those with the lowest scores) to most well off (those with the highest scores). Households ranked below the 30th percentile of the respective distribution are considered beneficiaries. This threshold approximately represents the actual share of beneficiaries in programme areas currently relying on HEA and PMT in Niger.

Simulating the PMT approach is straightforward because the PMT and ECVM/A survey instruments are harmonised, and the latter survey contains nearly all variables used in the PMT formula.¹¹ As a result, the PMT formula, together with a cut-off point, can be applied directly on the ECVM/A dataset to identify households that would be selected under PMT.

Simulating the HEA approach is more complex given the differences in survey instruments. Specifically, the HEA survey instrument tends to obtain information through a smaller number of questions relative to the ECVM/A survey. Specifically, among variables, including the period of coverage by own agricultural production (in months) and income, the HEA survey has one question for each indicator, while the ECVM/A survey has detailed modules that can be used to construct these indicators. Similarly, to obtain information about cultivated land, household size, number of children, and ownership of livestock, the ECVM/A survey relies on a larger set of more detailed questions. For each HEA variable used, Table 2 provides a detailed comparison between the HEA and the ECVM/A survey instrument. Relying on more detailed questions to construct the HEA score is likely to result in more precise targeting outcomes. The HEA performance outlined in this article should therefore be seen as an upper bound.

Table 2. Comparison of ECVM/A data with the HEA variables used for targeting

HEA variables	HEA survey instrument	ECVM/A survey instrument
Household size	Asks the respondent to report on household size. Normally, nongovernmental organisations use the following definition of household: a group of people living under the same roof and sharing the same pot of food.	Asks the name of each household member and then adds members to obtain the total household size. The survey defines the household as a group of persons, related or otherwise, living habitually in the same dwelling, pooling their resources, sharing their meals, and recognising the authority of the same person known as the household head. This question is asked during both the first and the second survey visits. The average household size of the two rounds is used.
Marital status of household head	Asks about the marital status of the household head. Household head is defined by the household, commonly the man or husband of the family or a woman who is a widow or divorced.	Asks about the marital status of the household head. Household head is defined as the person who is accepted as such by the other household members. The head generally exercises authority as the economic provider in the household. This question is asked during the first survey visit.
Number of poultry owned by the household	Asks about the number of poultry owned by the household.	Asks separately about the number of chickens, guinea fowl, and other birds the household owns. The number of animals in these categories are added up to construct the corresponding HEA variable. This information was only collected during the second survey visit.
Number of small ruminants owned by the household	Asks about the number of small ruminants the household owns. This includes goats, sheep, and donkeys.	Asks three different questions about the number of goats, sheep, and donkeys owned by the household. The number of animals in the responses is added up to construct the corresponding HEA variable. This information was only collected during the second survey visit.
Number of big ruminants owned by the household	Asks about the number of big ruminants the household owns. This includes cattle, camels, and horses.	Asks three different questions on the number of cattle, camels, and horses owned by the household. The number of animals in the responses is added up to construct the corresponding HEA variable. This information was only collected during the second survey visit.
Land surface cultivated by the household (hectares)	Asks a question about the total land surface cultivated by the household (hectares).	Asks about the area cultivated with each crop. The areas cultivated are added to construct the corresponding HEA variable. This information was only collected during the first survey visit.
The duration of food coverage using own agricultural production (months)	Asks one question on the duration of food coverage using own agricultural production (months).	The instrument includes a detailed food consumption module that asks about the consumption of various items during the seven days previous to the survey and about whether the food consumed was purchased or own produced. To construct the corresponding HEA variable, two variables constructed from the consumption module are used as follows: food consumption resulting from the consumption of own production, divided by total food consumption and multiplied by 12. Questions related to this indicator were asked during both the first and the second survey visits. The average of the indicator across the two seasons is used.
Average monthly household revenue	Asks one question on the average monthly household revenue (without specifying a time frame). This is defined as money earned (in cash only) by household members.	The instrument has a detailed module to capture sources of income during separate time frames depending on the question. Income from the following three sources are added up to construct the HEA variable: (a) employment income for each household member, (b) income from household enterprises, and (c) nonemployment income. This information was only collected during the first survey visit.

Notes: The table provides an exhaustive list of the variables used by HEA. The information used is based on the HEA methodological guide developed by the alliance (Bourahla et al., 2014). The way HEA questions are framed may slightly vary across the various nongovernmental organisations using HEA.

4.2. Efficiency of PMT and HEA

The study relies on inclusion errors to measure targeting efficiency. Inclusion errors are measured as the share of beneficiaries selected by the targeting methodology who are ineligible to participate in the programme. Eligibility is determined by the programme-eligibility threshold, that is, a household is eligible if it is ranked below the 30th percentile in the distribution of a given welfare metric. This threshold is largely similar to the rates of persistent poverty and transient food insecurity, the two main welfare metrics used in this report.

Based on the above definition of inclusion errors, exclusion errors – defined as the share of eligible households that are not selected by the targeting methodology – are exactly equivalent. This report only examines inclusion errors. The definitions imply that, for each wrongly included beneficiary (affecting inclusion errors), there is a wrongly excluded eligible household (affecting exclusion errors).

Table 3 presents inclusion errors based on different welfare metrics of a method that would randomly select beneficiaries (for benchmarking purposes), the PMT method, and the HEA method (columns 1, 2, and 3, respectively). Large and significant discrepancies are found between the PMT and HEA methods depending on the indicator and season considered.

Table 3. Inclusion errors based on selected welfare measures, %

Welfare measure	(1)	(2)	(3)	(4)	(5)	(6)
	Random	PMT	HEA	Diff. (2) − (3)	Diff. (1) − (2)	Diff. (1) − (3)
Consumption per capita, lean	70	50	72	−23***	20***	−2
Consumption per capita, harvest	70	44	71	−27***	26***	−1
Total consumption per capita	70	43	72	−29***	27***	−2
FCS, lean	70	70	52	18***	0	18***
FCS, harvest	70	65	61	4	5*	9***

Notes: Total consumption per capita represents the average consumption per capita across the two seasons. Inclusion errors are defined as the share of ineligible beneficiaries. The sample is restricted to beneficiaries (N = 243). A household is considered eligible if it is ranked below the 30th percentile in the distribution of the given welfare metric. All models assume the same number of beneficiaries (corresponding to 30 per cent of all households). FCS = food consumption score. Diff. = difference. p-values derived from tests of equality: * = 10 per cent, ** = 5 per cent, *** = 1 per cent.

Based on the FCS, HEA performs relatively well during the lean season; inclusion errors are fewer by 18 percentage points than the case of the PMT method or the random method (p-value < .01). In the case of consumption per capita in either season, PMT performs largely better than HEA or the random method. Indeed, HEA and the random method do not perform any differently with respect to each other in either season (p-value > .1). In contrast, PMT performs better by 27–29 percentage points than the random method or the HEA method based on total consumption per capita (p-value < .01). The results are consistent with the rather limited overlap between the beneficiaries selected by each method (see the Supplementary Appendix).

While the above results involve the selection of 30 per cent of households as beneficiaries, selecting different beneficiary shares can have large implications in targeting efficiency (see the Supplementary Appendix). Also, while the above results are based on binary indicators, it may be of interest to consider the performance of methods along the entire distribution (see the Supplementary Appendix).

4.3. Understanding the differences between PMT and HEA

Table 4 illustrates the characteristics of households selected by PMT and HEA. It shows that large differences are found in terms of household composition and livestock. PMT-selected households have 4.3 more household members, on average, than HEA households (p-value < .01). Only 11 per cent of PMT households are headed by women, versus 26 per cent of HEA households (the difference is 15 percentage points; p-value < .01). Polygamy is present in 39 per cent of PMT-selected households versus only 7 per cent among HEA-selected households (the difference is 33 percentage points; p-value < .01). Finally, PMT-selected households have at least twice as many small and large ruminants as HEA-selected households (the differences are, respectively, 1.9 percentage points [p-value < .01] and 0.5 percentage points [p-value < .01]).

Table 4. Characteristics of households selected by PMT and HEA

Indicator	(1)	(2)	(1) − (2)
	PMT	HEA	PMT − HEA
Household demographics
Household size	9.1	4.8	4.31***
Woman-headed household	0.11	0.26	−15%***
Polygamous family	0.39	0.07	33%***
Assets
Has cart	0.21	0.05	16%***
Has hoe	1	0.89	10%***
Number of small ruminants	2.9	1.1	1.9***
Number of large ruminants	0.5	0.0	0.50***
Cultivated land	17	5	12***
Non-Productive asset index	−0.46	−0.25	−0.22**
Other
Average monthly revenues in cash	28,068	13,957	14,111***
Food coverage based on own production (months)	3.5	2.0	1.5***
Number of sectors in which the household is engaged	2.5	1.9	0.64***
Share of income from remittances	9.2	21.1	−12.2%***

Notes: A household is considered eligible if it is ranked below the 30th percentile in the distribution of the given welfare metric. All models assume the same number of beneficiaries (corresponding to 30 per cent of all households). The sample is restricted to beneficiaries (N = 243). The non-productive asset index was constructed using principal component analysis based on selected variables, including dwelling characteristics and ownership of household goods such as telephone, beds, chairs, furniture, and so on. p-values derived from tests of equality: * = 10 per cent, ** = 5 per cent, *** = 1 per cent.

While PMT-selected households have significantly fewer non-productive assets (the difference is 0.22 percentage points; p-value < .05), HEA-selected households have significantly less revenue and narrower food coverage based on own production (the difference is, respectively, CFAF 14,111 [p-value < .01] and 1.5 months [p-value < .01]).

There are also important differences in productive activities. HEA-selected households have a smaller number of agricultural productive assets, such as carts or hoes (the difference is, respectively, 16 percentage points and 10 percentage points; both p-values < .01). They also have less diversified livelihoods, engaging in 0.6 fewer sectors than PMT households (p-value < .01).

Differences in characteristics suggest that households identified through HEA may be less able to cope with shocks than households identified through PMT. Smaller households with less livestock and less diversified livelihoods may be less capable of sharing risk across household members, less likely to rely on livestock as a buffer, or less likely to rely on alternative sources of revenues if a shock affects a particular livelihood activity. While differences are large in terms of the characteristics of households selected by PMT or HEA, this is not the case among those households that are wrongly excluded by each method (see the Supplementary Appendix).

There are important potential caveats in the application of PMT and HEA. Regarding PMT, time lags exist between the surveys used to develop the formula and programme implementation. Poverty correlates may change, which may cause the PMT formula to generate inaccurate predictions. Brown et al. (2017) find that implementation time lags matter considerably, while Premand and Schnitzer (2018) find otherwise. To assess this concern, the PMT formula was recalibrated based on the 2011 data. The results show that, in the context of this study, the timing of the PMT formula had a moderate effect, with fewer inclusion errors (by 7 percentage points), based on consumption per capita (see the Supplementary Appendix).

Regarding HEA, the weights of the HEA formula were qualitatively assigned based on the field experience of the nongovernmental organisation during several years. However, whether this formula can be easily and accurately replicated in other countries, whether it can be quickly updated, and the nature of the targeting efficiency impacts of the use of outdated formulas remain open issues. Also, while food coverage based on own production and revenues are likely to capture fluctuations in welfare more accurately relative to other variables used by both methods, they may also be more difficult to measure and verify. Finally, while the time to implement HEA may vary according to a range of factors (such as the size of the data collection team and the number of households to be covered), the process is relatively cumbersome. In Niger, the approach has usually been implemented during the response to slow-onset crises (such as those driven by droughts), and the ability to respond to a rapid-onset crisis may be a concern.

5. Geographical and combined targeting approaches

Geographical targeting is used widely in Niger, where it is often combined with household targeting approaches such as PMT and HEA. This section explores the performance of geographical targeting approaches used in isolation, as well as in combination with PMT and HEA. It first simulates two pure geographical approaches whereby areas are selected based on two distinct welfare indicators: persistent poverty and transient food insecurity. To select areas, primary sampling units were ranked according to the mean levels of each welfare indicator. Next, primary sampling units exhibiting the lowest level of welfare were selected. Within selected areas, every household was assumed to benefit from the programme. For comparability, the number of areas selected, which represents 30 per cent of all areas, was established to achieve a beneficiary quota equal to the one used under the PMT and HEA methods. Hereafter, the two simulated approaches are referred to as the geographical poverty method and the geographical food insecurity method, according to the welfare indicator applied.

This section also simulates two combined approaches relying on the geographical targeting approaches described above and in combination with PMT and HEA. Specifically, the geographical approaches were used to select half of all areas. Next, PMT and HEA were used to select households within areas suffering from persistent poverty and transient food insecurity, respectively. For comparison, the number of beneficiaries was set equal to the number used in the PMT and HEA methods. Hereafter, these two simulated approaches are referred to as the combined poverty method and the combined food insecurity method, respectively, according to the welfare indicator applied.

There are, however, limitations to the methodology applied to simulate the geographical methods. First, the validity of the results depends on the assumption that the welfare means obtained in primary sampling units are representative of the units and available for use. However, the survey was not designed to be representative at such a level of disaggregation, and, in practice, this information is not available at disaggregated geographical levels; only proxies of these measures may be available for use. Simulations thus provide an upper-bound estimate of performance. Second, while the analysis identifies areas in which food insecurity is currently being experienced, the objective should be to identify areas at risk of food insecurity, that is, before households become food insecure. The results presented in this report aim to illustrate only the potential of geographical targeting to address food crises; more research is needed to understand how this can be effectively implemented in practice.

The inclusion errors involved in the methods aimed at addressing poverty and food insecurity are presented in Table 5 (panels a and b, respectively). For comparison, the inclusion errors of the PMT and HEA approaches are also presented in each respective panel, as well as a universal approach whereby everyone is assumed to benefit from the programme.

Table 5. Inclusion errors, selected approaches, %

	(1)	(2)	(3)	(4)	(5)	(6)	(7)
Welfare measure	Universal	PMT	Geographical poverty	Combined poverty	Diff. (2) − (3)	Diff. (2) − (4)	Diff. (3) − (4)
a. Geographical and combined poverty targeting approaches
Consumption per capita, lean	70	50	58	48	−9*	1	10**
Consumption per capita, harvest	70	44	54	43	−10**	1	11**
Total consumption per capita	70	43	54	40	−11**	3	14***
FCS, lean	70	70	69	70	1	−1	−2
FCS, harvest	70	65	56	61	9**	4	−5
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
Welfare measure	Universal	HEA	Geographical food insecurity	Combined food insecurity	Diff. (2) − (3)	Diff. (2) − (4)	Diff. (3) − (4)
b. Geographical and combined food insecurity targeting approaches
Consumption per capita, lean	70	72	74	75	−1	−3	−2
Consumption per capita, harvest	70	71	68	64	2	7	4
Total consumption per capita	70	72	70	69	2	3	1
FCS, lean	70	52	40	43	12***	9**	−3
FCS, harvest	70	61	70	57	−9**	4	14***

Notes: Total consumption per capita represents the average consumption per capita across the two seasons. Inclusion errors are defined as the share of ineligible beneficiaries. The sample is restricted to beneficiaries (N = 243). A household is considered eligible if it is ranked below the 30th percentile in the distribution of the given welfare metric. Except for the universal model, all models assume the same number of beneficiaries (corresponding to 30 per cent of all households). FCS = food consumption score. Diff. = difference. p-values derived from tests of equality: * = 10 per cent, ** = 5 per cent, *** = 1 per cent.

The geographical poverty approach performs considerably better than a universal approach. The share of inclusion errors based on persistent poverty are 70 per cent in the universal approach, versus 54 per cent in the geographical poverty method. Nonetheless, compared with PMT or the combined approach, the geographical method performs significantly worse: the number of inclusion errors is 11 and 14 percentage points greater relative to the PMT or combined approach, respectively. Finally, the minor differences observed between the combined poverty method and PMT are not statistically different from zero.¹²

The food insecurity and poverty approaches present different patterns. In this case, a geographical food insecurity method performs especially well. Compared with a universal approach and the HEA approach, it reduces inclusion errors by 30 and 12 percentage points, respectively (based on the FCS during the lean season). This result is consistent with the fact that food insecurity is often experienced as a result of covariate shocks that affect a large share of households within given areas. The geographical food insecurity approach performs no different than the combined approach based on the FCS during the lean season. Also based on the FCS during the lean season, the combined food insecurity method performs better than HEA by 9 percentage points.

Geographical targeting is thus most effective if the relevant welfare measure presents both large variations across geographical units and little variation within geographical units. This is precisely why the performance of the geographical food insecurity approach is considerably better than the performance of the geographical poverty approach.¹³

6. A discussion about costs

This report focuses on the relative efficiency of various targeting methods, but costs should also be taken into account in choosing a targeting method. The costs should include not only administrative costs, but also other costs, such as private, incentive, social, and political costs (Coady, Grosh, & Hoddinott, 2004). The current data do not allow an exploration of all these costs. This section therefore briefly presents some costs associated with the targeting methods on which information is available and discusses potential implications.

According to administrative data from the Niger Safety Nets Project, applying the PMT questionnaire costs US$6.80 per screened household.¹⁴ Given that only 30 per cent of the screened households are selected, this corresponds to 5.5 per cent of the total transfers made per household during the two-year programme. The costs of applying the PMT questionnaire can be used as an estimate of the overall costs associated with the PMT method, but other factors would need to be taken into consideration to produce more precise cost estimates.¹⁵

The costs of applying the HEA questionnaire are unknown. Yet, they are likely to be similar to the application costs of the PMT questionnaire. However, the HEA method involves additional steps that may result in additional costs, including the categorisation of households by village committees, the triangulation of information between committees and survey-based results, and the correction of inconsistencies across the various sources of data.

Geographical poverty approaches usually rely on existing databases, and the costs are therefore only minor and mainly related to the labour costs of an analyst who can produce small area poverty estimates. Despite the targeting-related costs of geographical approaches, these approaches may generate overall project cost savings stemming from a reduction in implementation costs. This is especially the case in contexts where transport and communication costs are high.

If one considers the cost of inclusion errors as the full amount spent on noneligible beneficiaries (a high-bound cost estimate given the potential benefits going to noneligible beneficiaries), geographical poverty targeting would result in inclusion error–related costs that are 11.0 percentage points higher relative to PMT.¹⁶ Assuming that the administrative costs of the PMT method are 5.5 percentage points higher than the corresponding costs of geographical targeting (resulting from the administration of the PMT questionnaire), the net benefits of applying PMT relative to a geographical approach would be 5.5 percentage points (11.0 minus 5.5) of any given budget. In the case of HEA and assuming that administrative costs will be at least the same as the costs of PMT (5.5 per cent of total transfers), the net benefits of the application HEA relative to a geographical food insecurity targeting approach would actually be negative, given that there are no differences in the inclusion errors of the two approaches. While these estimates are based on an oversimplified scenario and limited data and while the geographical approaches provide upper-bound estimates of performance, the estimates provide a hint about the potential trade-offs among methods.

7. Conclusion

This report provides the first evidence on the relative performance of PMT and HEA. Errors are important, but PMT and HEA can more effectively identify households suffering from persistent poverty and transient food insecurity, respectively. Given the strong correlation between location and food insecurity, geographical targeting could be especially effective if the aim is to respond to food crises. However, existing early warning systems in the region suffer from various flaws, and their overall effectiveness is unknown. Research on cost-effective methods to identify locations at risk of food insecurity should be pursued, such us machine learning techniques, combined with satellite imagery.

In terms of the overall costs and benefits of the various methods, the results show that, in the context of the poor and largely homogeneous population that is the focus of this study, the net benefits of household-level targeting may be limited relative to the application of simple geographical approaches. The study is based on simulations; the actual performance of methods may differ.

Many low-income countries, such as Niger, face the dual challenges of persistent poverty and recurrent shocks. The choice of targeting method to address these challenges is a frequent topic in policy debates. This study illustrates that there is no single top-performing targeting method in absolute terms. Combinations of geographical, PMT, and HEA approaches may be considered to identify households suffering from persistent poverty and transient food insecurity as part of a scalable ASP system. The choice of method should depend on the programme-specific welfare objective, the distribution of welfare (across and within the targeted areas), costs, and the coverage rates achievable under binding budget constraints, leading to optimised programme impacts. Verme and Gigliarano (2019) provide a relevant example of a way to determine an optimal targeting strategy based on coverages rates, budgets, or poverty lines. Rather than focusing on identifying a single, optimal targeting method, actors trying to develop ASP systems might emphasise the consolidation of information across programmes through mechanisms such as a unified database.¹⁷ This could support more dynamic coordination and the effective application of alternative targeting methods.

Acknowledgements

Jacobus de Hoop, Amber Peterman, Ugo Gentilini, Tilman Brück, Jose Cuesta, Carlo del Ninno, Patrick Premand, Bradford Mills, Quentin Stoeffler, Arthur Alik-Lagrange, Phillippe Leite, Leila Bourahla, Marie Boulinaud, Michael Denly, Jennifer Ashley, and Julius Gunnemann provided valuable inputs and comments on the manuscript. Participants in workshops that took place in Washington, DC, and Niger provided useful comments. These workshops included the Food Economy Group and various nongovernmental organisations making use of the household economy analysis method. The research and the compilation of this work would not have been possible without the generous contribution of the Adaptive Social Protection Programme for the Sahel Multi-donor Trust Fund, funded by the United Kingdom. All data and Do files used in this study are available upon request.

Disclosure statement

No potential conflict of interest was reported by the author.

Supplementary Materials

Supplementary Materials are available for this article which can be accessed via the online version of this journal available at https://doi.org/10.1080/00220388.2019.1687877

Additional information

Funding

This work was supported by the Department for International Development.

Notes

1. Other productive assets include the hoe and the harrow.

2. For more information, see the HEA website (http://www.heawebsite.org/baseline-assessments).

3. The HEA approach used in Niger is described by Bourahla et al. (2014). Alliance ECHO is a partnership of nongovernmental organisations and United Nations agencies. Brück, Díaz Botía, Ferguson, Ouédraogo, and Ziegelhöfer (2019) describe examples of interventions in Niger relying on the HEA approach.

4. While HEA users in most other countries still rely on the CBT exercise only, there is an ongoing regional initiative to develop HEA in a systematic way.

5. Currently, 17 countries use the Cadre Harmonisé. See Food and Agriculture Organisation of the United Nations (FAO) (2019).

6. For more information, see National Institute of Statistics of Niger (INS-Niger) and World Bank (2013).

7. The study also considers alternative welfare indicators (see the Supplementary Appendix).

8. The food groups include cereals and cereal products, tubers and plantains, legumes and seeds, vegetables, fish and meat, fruits, milk and milk products, oil and grease, sugar products, and spices and condiments.

9. A household is considered poor if its annual consumption per capita is below the national poverty line threshold (CFAF 182,635). A household is considered food insecure if its FCS is below 35. These definitions follow the definitions used by the National Institute of Statistics of Niger (INS-Niger) and World Bank (2013).

10. This outcome does not necessarily mean that the poverty perceptions of communities are the same as those resulting from the PMT approach. Anecdotal evidence from the field suggests that, because of cultural norms, communities tend to avoid disagreeing with the list of selected beneficiaries provided during the validation process.

11. An exception is the presence of a disabled or handicapped individual in the household, which is thus excluded from the PMT formula that is simulated in this article.

12. In the application of a geographical poverty approach, only proxies of consumption, as opposed to consumption itself, are likely to be available in small geographical areas. To explore this concern, the study also relied on a proxy of consumption (the PMT scores) to simulate geographical poverty targeting showing that results do not significantly change (see the Supplementary Appendix).

13. The tables in the Supplementary Appendix show that the intracluster correlation of transient food insecurity (intracluster correlation = 0.37) is substantially larger than the intracluster correlation of poverty (0.09).

14. The estimates are based on variable costs, including all field staff–related costs and logistics, and excluding fixed costs that are linked to multiple programme aspects other than targeting, such as government administrative costs.

15. For example, the PMT data gathered will likely result in cost savings during the beneficiary registration process. However, the development of a PMT formula and questionnaire will result in additional (but minor) costs, mainly related to the labour costs of an analyst.

16. The estimate is based on the fact that geographical targeting results in 9 percentage points more in inclusion errors relative to PMT (see Table 3).

17. For a more detailed discussion on building a unified database, see the Supplementary Appendix.