Evidence in Demand: An Overview of Evidence and Methods in Assessing Impact of Economic and Social Rights

Abstract

The article examines evidence and methods in the recent human rights literature which reports on the impact of economic and social rights. The article is structured according to the four main analytical entry points under which distinct methods of social rights impact assessment are used: social rights litigation (evidence through analysis of legal texts and mainly qualitative evidence), the results of ratifications and constitutionalisation (evidence through comparative quantitative data analysis), the results of human rights policy reforms (evidence through the use of quantitative data and qualitative interpretation) and the results of civil society advocacy and human rights-based approaches (predominantly qualitative evidence). The paper concludes that positive changes are documented with respect to all methods examined, but the causal linkages between the social rights analysed and the results on the ground are not always well-established, or they provide conflicting interpretations, demonstrating that this field of research is still relatively young.

Keywords:

• Impact Economic
• Social and Cultural Rights Litigation
• Policy Reform
• Ratifications
• Cross-Country Comparisons
• Advocacy and Empowerment

Notes

¹ T Landman and E Carvalho, Measuring Human Rights (Routledge, London, 2010). EM Hafner-Burton and J Ron, “Seeing Double: Human Rights Impact through Qualitative and Quantitative Eyes” (2009) 61(2) WorldPolitics, 360. See also EM Hafner-Burton, “A Social Science of Human Rights” (2014) J of Peace Research 51(2), 273. F Coomans, F Grünfeld and MT Kamminga (eds), Methods of Human Rights Research (Intersentia, Antwerp, 2009).

² No distinction is made in this article between outcomes and impact. In programme implementation and evaluation, outcomes are associated with the nature of positive change produced by the programme (programme effectiveness), whereas impact relates to the longer-term intended or unintended negative or positive changes that are attributable to the programme. See EA Andersen and H-O Sano, Human Rights Indicators and the Program and Project Level (Danish Institute for Human Rights, Copenhagen, 2006). However, outside the specific programme frames, outcomes and impact are often blurred. UNHCHR, Human Rights Indicators, A Guide to Measurement and Implementation (UNHCHR, Geneva, 2012) classifies one set of indicators as “Outcome indicators” without making distinctions between outcomes and impact.

³ See H-O Sano and H Thelle, “The Need for Evidence-Based Human Rights Research”, in F Coomans, F Grünfeld and MT Kamminga (eds), Methods of Human Rights Research (Intersentia, Antwerp, 2009).

⁴ Landman and Carvalho treat administrative data in their discussions on sources used in events-based analysis and thus link such data to human rights violations. See T Landman and E Carvalho, Measuring Human Rights (Routledge, London, 2010), 36–39 and 51–53). However, administrative records may also be policy documents from governments and are therefore not only used in event-based analyses.

⁵ Documentary evidence could be government publications and legislation; non-government institutional sources such as UN documents or documents produced by NGOs; academic publications; printed media and visual documentation. Reed et al label this methodological entry “Document Review”. See K Reed and A Padskocimaite, The Right Toolkit. Applying Research Methods in the Service of Human Rights” ( Human Rights Center, University of California, Berkeley, 2012), 12–14.

⁶ For a predominant focus on compliance, see special issue of Nordic Journal of Human Rights 2012, vol 3, edited by M Langford and S Fukuda-Parr.

⁷ The term of “legal reflex” was used by P Gready and J Ensor in their Introduction to Reinventing Development? Translating Rights-Based Approaches from Theory into Practice (Zed Books, London 2005). They described legal reflexes as the automatic and unconsidered recourse to legal instruments to ensure protection and promotion of human rights (9).

⁸ This is the broad concept used by authors such as S Engle Merry, “Rights Talk and the Experience of Law. Implementing Women's Human Rights to Protection from Violence” (2003) 25(2) Human Rights Quarterly, 324. See also V Gauri and S Gloppen, “Human Rights-Based Approaches to Development: Concepts, Evidence, and Policy” (2012) 44 Polity 496, 502.

⁹ A Sen identified the paradox of micro-successes and macro-failures in poverty reduction in Development as Freedom (Alfred A. Knopf, New York, 1999). See also International Council on Human Rights Policy, “No Perfect Measure: Rethinking Evaluation and Assessment of Human Rights Work. Report of a Workshop” (2012, Geneva, Switzerland). Available at http://www.ichrp.org/files/reports/68/181_evaluating_hr_work_report.pdf (accessed 3 February 2014).

¹⁰ For an example of this, see WM Cole and FO Ramirez’ discussion on the impact of national human rights institutions: “Conditional Decoupling: Assessing the Impact of National Human Rights Institutions, 1981to 2004” (2013) 78(4) Am Soc Rev, 702. Cole and Ramirez report on a particular dimension of decoupling with departure in what Keck Sikkink has termed the information paradox: the fact that human rights observers on the ground expouse perceptions of deteriorating human rights practices even when objective practices remain constant due to the fact that changing conditions, for instance newly established national human rights institutions exposing information on hitherto unknown human rights violations (708).

¹¹ EM Hafner-Burton and J Ron, “Seeing Double: Human Rights Impact through Qualitative and Quantitative Eyes” (2009) 61(2) WorldPolitics, 363.

¹² K Reed and A Padskocimaite, The Right Toolkit. Applying Research Methods in the Service of Human Rights” (Human Rights Center, University of California, Berkeley, 2012), 9.

¹³ UNHCHR, Human Rights Indicators, A Guide to Measurement and Implementation (UNHCHR, Geneva, 2012). Available at http://www.ohchr.org/EN/Issues/Indicators/Pages/HRIndicatorsIndex.aspx (accessed 6 January 2014); S Mcinerney-Lankford and H-O Sano, Human Rights Indicators in Development. An Introduction (World Bank, Washington DC 2010).

¹⁴ T Landman uses the term standard-based data from the coded country level information about (mostly) civil and political rights on a standardised scale that typically is both ordinal and limited in range. Examples are the CIRI Human Rights Data Project, the Freedom House Civil and Political Liberties scale or the Political Terror Scale. See T Landman and E Carvalho, Measuring Human Rights (Routledge, London, 2010), 64.

¹⁵ M Langford, “Social Rights Jurisprudence: Emerging Trends in International Comparative Law” (Cambridge University Press, Cambridge, 2008).

¹⁶ V Gauri and S Gloppen, “Human Rights-Based Approaches to Development: Concepts, Evidence, and Policy” (2012) 44 Polity 496–, 496–502.

¹⁷ V Gauri and S Gloppen, “Human Rights-Based Approaches to Development: Concepts, Evidence, and Policy” (2012) 44 Polity 496–, 497.

¹⁸ Gauri and Brinks reviewed court cases in Nigeria on the right to education and health and noted a timidity of courts in Nigeria concerning social rights unlike the situation in Brazil, India and Indonesia. In Nigeria the courts were more concerned with the rights in education compared with the rights to education, Brinks and Gauri, “Introduction: The Elements of Legalization and the Triangular Shape of Social and Economic Rights” in V Gauri and DM Brinks (eds), Courting Social Justice (Cambridge University Press, Cambridge, 2008, 8–9.

¹⁹ DM Brinks and V Gauri, “A New Policy Landscape: Legalizing Social and Economic Rights in the Developing World” in V Gauri and DM Brinks (eds), Courting Social Justice (Cambridge University Press, Cambridge, 2008), 334–351.

²⁰ DM Brinks and V Gauri, “A New Policy Landscape: Legalizing Social and Economic Rights in the Developing World” in V Gauri and DM Brinks (eds), Courting Social Justice (Cambridge University Press, Cambridge, 2008), 340–341.

²¹ See V Gauri and DM Brinks, “Introduction: The Elements of Legalization and the Triangular Shape of Social and Economic Rights” in V Gauri and DM Brinks (eds), Courting Social Justice (Cambridge University Press, Cambridge, 2008, 22–25.

²² In estimating the number of people benefitting directly and indirectly from two collective court cases on antiretroviral treatment medicine in India, Gauri and Brinks used a formula that involves estimates by Indian researchers based on qualitative research on the proportion of benefits that actually reached the intended recipients. For the details, see V Gauri and DM Brinks, Courting Social Justice (Cambridge University Press, Cambridge, 2008), 226–27.

²³ In the following section, I use “Changing Human Rights Structures” to denote changes in the acceptance, commitment and intent to undertake measures in keeping with human rights obligations. This also involves the incorporation of human rights law into domestic law. See UNHCHR, Human Rights Indicators, A Guide to Measurement and Implementation (UNHCHR, Geneva, 2012), 34.

²⁴ These studies had either negative conclusions on the effect of treaty ratifications (O Hathaway, “Do Human Rights Treaties Make a Difference?” (2002) 111 Yale Law Rev J 1935–2042) or they provided very nuanced or negative conclusions, for instance E Neumayer, “Do International Human Rights Treaties Improve Respect for Human Rights?” (2005) 49 (6) J of Conflict Resolution, 925–53.

²⁵ W Cole, “Strong Walk and Cheap Talk: The Effect of the International Covenant on Economic, Social and Cultural Rights” (2013) 92 (1) Social Forces 167.

²⁶ In fact, US scholarship dominates this quantitative line of research focusing on regression analyses. Exceptions are E Neumayer's “Do International Human Rights Treaties Improve Respect for Human Rights?” (2005) 49 (6) J of Conflict Resolution. Also C Bjørnskov and J Mchangama, Do Social Rights Affect Social Outcomes? (Social Science Research Network, Copenhagen, September 2013) is available at the network. Neither of these studies are reviewed here, given that the former dates back some time, and the latter has not been published in a peer-reviewed journal.

²⁷ See W Cole, “Strong Walk and Cheap Talk: The Effect of the International Covenant on Economic, Social and Cultural Rights” (2013) 92 (1) Social Forces 167, 171–72.

²⁸ W Cole, “Strong Walk and Cheap Talk: The Effect of the International Covenant on Economic, Social and Cultural Rights” (2013) 92 (1) Social Forces 167, 180 and 187.

²⁹ B Simmons, Mobilizing for Human Rights. International Law in Domestic Politics (Cambridge University Press, Cambridge, 2009), 159–255.

³⁰ B Simmons, Mobilizing for Human Rights. International Law in Domestic Politics (Cambridge University Press, Cambridge, 2009), 200–2.

³¹ B Simmons, Mobilizing for Human Rights. International Law in Domestic Politics (Cambridge University Press, Cambridge, 2009), 215.

³² B Simmons, Mobilizing for Human Rights. International Law in Domestic Politics (Cambridge University Press, Cambridge, 2009), 217. It should be noted that Simmons, among those who have analysed empirically the influence of treaty ratification, is deemed “probably the most optimistic”. See V Gauri and S Gloppen, “Human Rights-Based Approaches to Development: Concepts, Evidence, and Policy” (2012) 44 Polity 496–, 491.

³³ A Palmer, J Tomkinson, C Phung, N Ford, M Joffres, KA Fenandes, L Zeng, V Lima, JSG Montaner, GH Guyatt and EJ Mills, “Does Ratification Of Human Rights Treaties Have Effects On Population Health?” (2009) 373(9679) The Lancet 1987–92.

³⁴ L Scruggs, C Zimmermann and C Jeffords, “Implementation of the Human Right to Social Security Around the World: A Preliminary Analysis of National Social Protection Laws”, in L Minkler (ed), The State of Economic And Social Human Rights. A Global Overview (Cambridge University Press, Cambridge, 2013), 117–134.

³⁵ Apart from the estimated importance of ratifying the covenant, Scruggs, Zimmermann, and Jeffords conclude that country income levels play a role together with socialist legal traditions. Uncertain explanatory variables are democracy and ethnic heterogeneity. See L Scruggs, C Zimmermann and C Jeffords, “Implementation of the Human Right to Social Security Around the World: A Preliminary Analysis of National Social Protection Laws”, in L Minkler (ed), The State of Economic And Social Human Rights. A Global Overview (Cambridge University Press, Cambridge, 2013), 131.

³⁶ L Scruggs, C Zimmermann and C Jeffords, “Implementation of the Human Right to Social Security Around the World: A Preliminary Analysis of National Social Protection Laws”, in L Minkler (ed), The State Of Economic And Social Human Rights. A Global Overview (Cambridge University Press, Cambridge, 2013), 132.

³⁷ Vandenhole and Gready point this out in the present volume. J Oestrech has conducted three case studies on how international organisations (WHO, UNICEF, and the World Bank) incorporate and deal with human rights. See J Oestrech, Power and Principle. Human Rights Programming in International Organizations (Georgetown University Press, Washington DC, 2007).

³⁸ OECD and World Bank, Integrating Human Rights into Development. Donor Approaches, Experiences, and Challenges, 2nd edn (OECD and the World Bank, Washington DC, 2013). See also D D'Hollander, A Marx and J Wouters, “Integrating Human Rights in Development Policy. Mapping Donor Strategies and Practices” (Leuven Centre for Global Governance Studies, Leuven, 2013), Paper #4.

³⁹ F Bustreo and P Hunt, “Women's and Children's Health: Evidence of Impact on Human Rights” (World Health Organization, 2013). Available at http://apps.who.int/iris/bitstream/10665/84203/1/9789241505420_eng.pdf (accessed 3 February 2014).

⁴⁰ JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology.

⁴¹ JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology, 13, Bustreo and Hunt 88-89.

⁴² Distinctions are made between historical control groups (same target institutions or population, Internal control groups (individuals or groups who should have received the intervention, but did not), and External control groups. See JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology, 11–15.

⁴³ JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology, 14.

⁴⁴ It can be argued that too little is done to falsify the thesis of positive human rights impact.

⁴⁵ JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology, 40.

⁴⁶ JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology, 43–47.

⁴⁷ JP Habicht, CG Victoria and JP Vaughan, “Evaluation Designs for Adequacy, Plausibility and Probability of Public Health Programme Performance and Impact” (1999) 28(1) Int J of Epidemiology, 97.

⁴⁸ See H-O Sano, “The Drivers of Human Rights Change in Development”, in P Gready and W Vandenhole (eds), Human Rights and Development in the New Millennium. Towards a Theory of Change (Routledge, Oxon, 2014), 37.

⁴⁹ See D D'Hollander, A Marx and J Wouters, “Integrating Human Rights in Development Policy. Mapping Donor Strategies and Practices” (Leuven Centre for Global Governance Studies, Leuven, 2013), Paper #4, 33, quoting Kindornay.

⁵⁰ For national importance in Kenya, see BA Andreassen, “Legal Empowerment of the Poor – a Strategy for Social Change”, in in P Gready and W Vandenhole (eds), Human Rights and Development in the New Millennium. Towards a Theory of Change (Routledge, Oxon, 2014), 101–2.

⁵¹ V Gauri and S Gloppen, “Human Rights-Based Approaches to Development: Concepts, Evidence, and Policy” (2012) 44 Polity 496–, 486–87 use a broader concept of HRBA as “targeted interventions on part of governments, donors and civil society actors”.

⁵² R Mutatkar, “State, Civil Society And Justice: The Case of India”, in M Albrow and H Seckinelgin (eds), Global Civil Society 2011. Globality and the Absence of Justice (Palgrave Macmillan, Houndmills, 2011).

⁵³ The excellent paper by D D'Hollander, A Marx and J Wouters, “Integrating Human Rights in Development Policy. Mapping Donor Strategies and Practices” (Leuven Centre for Global Governance Studies, Leuven, 2013), Paper #4 does not address local level impact in detail.

⁵⁴ K Young, Constituting Economic and Social Rights (Oxford University Press, Oxford, 2012), 223, 232.

⁵⁵ BA Andreassen, “Legal Empowerment of the Poor – a Strategy for Social Change”, in in P Gready and W Vandenhole (eds), Human Rights and Development in the New Millennium. Towards a Theory of Change (Routledge, Oxon, 2014), 100.

⁵⁶ A Sarelin, Exploring the Role and Transformative Potential of Human Rights in Development Practice and Food Security. A Case Study from Malawi (Åbo Akademi University Press, Åbo, 2012).

⁵⁷ N Kabeer, A Haq Kabir and T Yasmi Huq, “Quantifying the Impact of Social Mobilisation in Rural Bangladesh: Donors, Civil Society and the ‘Road not Taken’”, IDS Working Paper 333 (2009). Available at http://www.ids.ac.uk/files/dmfile/Rs3331.pdf (accessed 3 February 2014).

⁵⁸ M Björkman and J Svensson, “Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project” (2008) 124(2) QuarterlyJ of Economics 735–, 769. See also M McNeil and C Malena (eds), Demanding Good Governance. Lessons from Social Accountability Initiatives in Africa (World Bank, Washington, DC, 2010).

⁵⁹ D Ringold, A Holla, M Koziol and S Srinivasan, Citizens and Service Delivery. Assessing the Use of Social Accountability Approaches in Human Development (World Bank, Washington, DC, 2012).