Challenges of research in rural poverty: lessons from large field surveys

ABSTRACT

This practical note highlights lessons learnt during the data collection of two large field surveys as part of the ongoing “Sindh Union Council and Economic Strengthening Support” (SUCCESS) programme in southern Pakistan. The experience is discussed in terms of language barriers, the educational status of households, dealing with people’s expectations and non-cooperation, and the weather conditions. The note also highlights the practice of public sharing of data in real-time to improve the design and implementation of future surveys, especially those measuring poverty and quality of life.

KEYWORDS:

• Aid – Monitoring and Evaluation
• Capacity development
• Methods
• Labour and livelihoods – Poverty reduction
• South Asia

Introduction

To eradicate extreme poverty by 2030, in line with the United Nations’ Sustainable Development Goals (United Nations 2015), one key challenge is access to authentic data about the status of poor households and the progress they make over time. This is easier said than done, since several factors impede the implementation of the different tools for data collection, processing, and analysis. In this practical note we share the lessons learnt from two surveys conducted in rural Sindh as part of a large-scale community development programme. Funded by the European Union, the Sindh Union Council and Economic Strengthening Support (SUCCESS) Programme builds on the experiences of the Union Council Based Poverty Reduction Programme of the Government of Sindh, which took place in 2009–11. Taking it a step further, SUCCESS supports the provincial government in developing its local community-driven development (CDD) policy, allowing a wider geographical outreach and providing financial means to reduce poverty in rural Sindh (Rural Support Programmes Network 2017). The programme aims to improve the living conditions of rural populations by building local social capital for better access to basic social and economic services, and providing means for income-generating and diversification activities. The programme was rolled out in October 2015 by the Rural Support Programmes Network (RSPN) and its three-member RSPs in Sindh: the Sindh Rural Support Organisation, Thardeep Rural Development Programme, and National Rural Support Programme. It spans eight out of the 24 districts in Sindh province for a period of six years, with an overall budget of EUR 82.13 million.

As part of the data collection exercise, two large surveys were carried out in 2016. First, the poverty score card (PSC) was undertaken in the eight programme districts; second, a socio-economic baseline survey was carried out in two union councils of Tando Allahyar district.

PSC as a tool for “poverty targeting” rather than poverty measurement was mainly developed to give development practitioners a simple, efficient, and cost-effective method for identifying a representative section of the population and conducting targeted programme interventions. For PSC we used the proxy means test (PMT) formula, derived from Pakistan Standard Living Measurement Survey (PSLM) 2007, with a set of easily verifiable indicators that correlate with the situation of poverty along with a simple scoring system (Hou 2011; World Bank 2013). This extensive exercise took place across all the eight programme districts, covering 316 union councils, 1,545 villages, and 14,706 settlements. This amounted to data collection for a population of 5,691,953 people in 849,754 households.

The socio-economic baseline survey was concentrated in two union councils randomly sampled in the rural area of Tando Allahyar district. This survey covered a population of 14,822 people: 7,667 males and 7,155 female respondents from 2,298 households.

In order to conduct the surveys, two separate questionnaires were developed, sampled, and pilot-tested in the English language. The preference for English was to facilitate feeding the responses from interviewees directly into the database. The first set of questions was termed a “general questionnaire”, which comprised questions to be answered by the heads of the households, whether male or female. The other set of questions, the “female questionnaire”, was specifically designed for female members of the household. Each questionnaire was projected to take about 45 to 90 minutes depending upon the family size, interview location, and the number of interruptions during the survey. The survey was conducted in the summer months of June to August 2016. The initial part of the recruitment, training, and test launch coincided with Ramadan, the month of fasting, starting in the first week of June. Young females were locally hired as enumerators. These enumerators were given an additional three to five days’ training on the process and conduct of the questionnaires, and how to translate interviewee responses and feed the comments into the database using tablet computers. Use of tablets helped with the geotagging of all inputs so that the researchers could contextualise the physical location and proximities of the data collection and interviewing processes.

Challenges and observations

The large-scale survey and research exercise work produced a number of opportunities as well as challenges for the enumerators and researchers. These ranged from minor everyday problems and obstacles accessing the respondents, to larger existential issues on the need, validity, and verification of the whole process. Lack of sufficient infrastructure also meant that the enumerators had to be taken to more remote and secluded areas using dedicated transport. We discuss some of the more interesting challenges and observations below.

Language barriers

Being an internationally funded development and research project, all information processing had to be carried out in English, and the questionnaires were developed in English, so that the collected data would be immediately available for observation and analysis by all the parties involved in the management and conduct of the programme. However, most enumerators found it difficult to fully understand and contextualise the open-ended questions, due to their limited language fluency. All enumerators were local recruits with a minimum qualification of bachelors’ degrees, and masters’ degrees in disciplines such as sociology, political science, and economics. This, however, did not guarantee a capacity to translate English questions into Sindhi or to eloquently paraphrase the original responses from Sindhi into English for documentation purposes.

Initially only native Sindhi-speaking females were recruited for enumeration. However, with a high number of dropouts due to extremely hot weather, the research team decided to reward and retain the experienced, qualified, and high-performing enumerators. In many cases, enumerators’ pay was revised upwards and the condition of native Sindhi speaker was removed to attract more female enumerators. This strategy succeeded in enrolling more enumerators. The language barrier potentially affected data quality, particularly about household income sources, since the enumerators could not extract precise information about all the sources of income. Upon analysis, nearly half the respondents appeared to have no income whatsoever. The researchers decided to use expenditure data from the survey as a proxy for calculating poverty and other income-related indicators.

Managing expectations and getting correct responses

Besides language difficulties on the part of the enumerators, most of the respondents continued to under-report their income, with many reporting their income levels as zero during the socio-economic baseline survey. One key factor in this respect was assumed to be the expectation of monetary benefits. Despite the previous experience of RSPs in their work with rural communities, obtaining the correct responses from respondents proved challenging. Though the selected sites did not have RSPs present, these communities experienced a huge influx of aid organisations in the aftermath of the 2010 floods. While the enumerators repeatedly told the interviewees that the survey was meant to assess the status of the community and no immediate benefits should be expected, the communities had a strong impression of some ensuing beneficial package. However, over 50% of respondents avoided reporting their income sources, despite a higher incidence of annual expenditures. For example, even if the household owned the livestock in the dwelling, they reported it to be on a shared basis with the landlords.

Through concurrent monitoring, the GIS coordinates were plotted on Google Maps to observe unusual clustering of households. Leveraging technology and use of computer assisted personal interviews (CAPI) helped to improve the data quality and reliability.

Dealing with non-cooperation

In some situations, especially during the PSC implementation, some communities refused to allow the enumerators to enter their houses. Three reasons can be attributed to such behaviour: purdah, previous bad experiences, or the social status of the household. In the first instance, some of the villages where purdah is observed by women would restrict male access, while allowing females access. However, there were a few cases where some households refused to cooperate and had to be exempted from the survey. Second, some villages had previous bad experiences of cold-callers and fake lottery schemes, where outside people came and defrauded people of their money by promising large cash and in-kind rewards. To overcome this trust issue, influential local people including activists and their families were taken into confidence about the surveys’ purpose. Social assurance was given to those unsure about the authenticity of the survey activity. The third category involved well-off households who considered the project as a charity programme. Due to their higher social status, they refused to cooperate on the grounds of not needing any charity or aid. However, once it was explained, some of the households were willing to participate. In all cases where households persisted with their decision not to participate in the surveys, their choices were respected.

Lack of education

Self-reporting data collection can be challenging for non-literate, remote, and rural communities due to minimal documentation available for income, expenditure, and savings at the household level. Over 90% of households reported that their land income and expenditure accounts were maintained by the local agents of the landlords, and they had only a rough idea of their earnings from the land. In the socio-economic baseline survey, 6,963 adults were surveyed, of which 1,649 were literate. Nearly a quarter of the sample households (23.8%) were literate male adults and 23.5% were literate female adults. Among the literate adults, almost two-thirds (63.6%) had completed primary education, and 14.9% had attended middle school. Hence, the enumerators were trained to use the triangulation tool (Yin 2003) in order to verify the information given.

Initially, both male and female enumerators were hired. However, only females were retained as it became difficult to have them jointly working in teams and sharing the remuneration. Remuneration was determined on the basis of each completed form. Since a major part of the data collection (the female questionnaire) had to be filled with a female adult member of the household, the researchers decided to work with female enumerators only. This, however, required additional logistical arrangements to and from the urban area (Hyderabad) to the research site in Tando Allahyar, a 120 km return journey taking about four hours. During the monsoon season, the daily commute gave rise to another set of transport issues, such as traffic jams due to flash flooding.

Weather conditions

Monsoon rains were not the only weather variable, and other seasonal biases affected the survey process. According to Chambers (1997, 2005), to gain true insight, development fieldwork needs to capture peoples’ responses in both pleasant and unpleasant weather conditions. To have a better understanding of the quality of life, the first baseline survey was started in the extremely hot season from June to August. In the field, the enumerators recorded respondents’ views under tree shade. With more than 10 hours of work in the rural areas in those days coinciding with Ramadan, it was challenging to follow Chambers’s advice of conducting fieldwork in this unpleasant situation. Those respondents and enumerators who observed fasting in Ramadan found it difficult to engage with the survey.

Another weather-related issue was the respondents’ boredom and enumerators’ fatigue. In some cases, certain sections of the questionnaire were being returned unfilled and thus treated in “no response” category in the analysis, in particular, the sections about household income, assets, savings and loans, which were therefore excluded from the overall analysis. To overcome this issue in future, the questionnaires were revisited and shortened to make sure that on average they do not take more than 45 minutes to complete. Future surveys are also being planned to avoid the fasting month and extreme weather conditions, in order to capture the reality of rural poverty and the seasonal variations.

Public sharing of data

In development practice, different datasets are prepared by practitioners to design a variety of project-related activities. However, these datasets are rarely shared during the project’s lifetime, thus restricting the use of the information by other institutions. For the SUCCESS programme, the team adopted an open approach to data sharing through a dedicated data portal providing real-time access to the PSC and collected data on the rural households in the eight programme districts of the Sindh province. Every month, updated data are broadcasted to a large number of organisations, individuals, and affiliates who could be potential users of this information.

Lessons learnt and course correction

From the challenges and observations experienced during the course of the first major phase of large-scale data collection and field survey, a number of lessons were learnt that allowed for course adjustment and improvement in the overall approach to development and research in extremely poor and distant rural communities. Language became one of the critical barriers in the communication of objectives and questions in order to get precise feedback. Translation between different languages (English, Urdu, and Sindhi) was a potential source of data anomalies. The research team (having multilingual skills) had to make regular adjustments to the strategy in order to make the best use of opportunities and build long-term relations with the communities, while at the same time providing safeguards and peace of mind to their teams of enumerators and field personnel. Coupled with language was the question of trustworthiness. The team tried to avoid the perception of development researchers as one-off visitors to deprived communities, collecting data, distributing cash rewards, and then wrapping up the project. With a strategic direction and long-term development objectives of SUCCESS, the team remained committed with the communities and built lasting relations with community representatives, leaders, and individual households. This also helped in minimising the ambiguity in responses and allowed analysts to contextualise the lack of information in parts of the survey through that provided in other parts (e.g. income versus expenses). The team treated respondents with dignity and respected the preferences of the households, especially women, to choose a place to undertake the interview where they would feel comfortable. We also appreciated the decisions of some households not to participate, once all communication channels were exhausted. Education emerged as a critical aspect, where the lack of decent education meant that people were less aware of their rights and responsibilities. In circumstances where respondents happen to be uneducated or appeared to withhold particular pieces of information, triangulation tools proved an efficient way to fill gaps in the information provided by respondents. This technique is also useful when respondents are relying on their memory recall.

Extreme weather conditions proved a bigger challenge in the fieldwork and should be given due consideration in prior planning. Respondents’ fatigue due to very hot weather and long hours may cause a lack of attention and thus affect data quality. Similarly, time of the day can also affect respondents’ moods and thus their responses. Pre-visits to field areas can help with arranging appropriate times of the day beforehand.

Leveraging technology through the use of CAPI, concurrent real-time monitoring, and correction and cleaning of data during the data collection process, as practiced in SUCCESS, is another precedence to be followed by development organisations.

Conclusion

In the longer term, large-scale implementation programmes should give due consideration to the dynamic nature and consequent changes in the target area, and these addressed in order to reduce the chances of unintended negative outcomes. In developing country settings, contextual factors and researchers’ biases could play out in the implementation of research surveys. In such situations, sometimes it is feasible to outsource the survey work to professional researchers who may have wider experiences of undertaking large field surveys in the given or similar areas. Under this assumption, for the next step, SUCCESS will experiment with outsourcing the second stage of socio-economic surveys to third parties, such as academic institutions and professional survey firms. The questionnaires will be made available in Sindhi.

Acknowledgements

The Sindh Union Council and Community Economic Strengthening Support Programme (SUCCESS) is funded by the European Union. The opinions in this paper are those of the authors only and can in no way be taken to reflect the views of the Rural Support Programmes Network, European Union or their partners.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Abdur Rehman Cheema is a Team Leader Research at the Rural Support Programmes Network, Islamabad, Pakistan. His areas of interest are the interconnected issues of socio-economic development, including governance, education, agribusiness, environment and climate change, and of regional developments, such as the China Pakistan Economic Corridor (CPEC).

Abid Mehmood is a Research Fellow at the Sustainable Places Research Institute, School of Social Sciences, Cardiff University, UK. His broader research expertise is in social innovation and governance. He has experience of research in climate change, socio-economic development, and social cohesion for local and regional development policy and practice.

Fazal Ali Khan is a Programme Manager of the SUCCESS programme at the Rural Support Programmes Network, Islamabad, Pakistan. He is a development professional with experience in designing and conducting large-scale surveys, planning, monitoring and evaluation of development programmes, and managing internal and external research and evaluations.

ORCID

Abdur Rehman Cheema http://orcid.org/0000-0003-0945-1986

Abid Mehmood http://orcid.org/0000-0003-3719-6388

Additional information

Funding

The Sindh Union Council and Community Economic Strengthening Support Programme (SUCCESS) is funded by the European Union.