Lost in operationalisation: developing legally relevant indicators, questions and benchmarks

ABSTRACT

The main purpose of this article is to provide methodological guidance to students and researchers conducting quantitative studies on the realisation (outcome) of human rights law. The article focuses on the grey area between law and sociology. How to ensure a high score on legal relevance in quantitative socio-legal studies on the realisation of human rights law. To discuss this two disciplinary traditions are merged – sociology of law and human rights research on indicators and benchmarks. A key tool for making legally relevant questions in surveys are indicators. Benchmarks are also important for some provisions to determine if an obligation has been fulfilled by the state. The process of translating the law to indicators, questions and benchmarks is called operationalisation. Each operationalisation process must aim for a high score on legal relevance and it should be transparent and open to criticism. To first spend a lot of time and money on surveys and data collection, and then to miss the law one is pretending to measure is not just embarrassing, but hazardous to the protection of human rights, because people then have a tendency to rationalise and present alternative facts about the law.

KEYWORDS:

• Operationalisation
• indicators
• surveys
• validity
• sociology of law

Disclosure statement

No potential conflict of interest was reported by the author.

Note on contributor

Hadi Strømmen Lile is an associate professor at Østfold University College in Norway. He got his PhD from the law faculty at the University of Oslo. He has been teaching sociology of law and human rights law for many years. He has been an advisor for the Gáldu – The Resource Centre for the Rights of Indigenous Peoples and a senior advisor for Save the Children Norway. He is the chief editor of the journal Critical Law (Kritisk juss) in Norway.

Notes

1 Martin Luther King, Where Do We Go from Here: Chaos or Community (Boston: Beacon Press, 1968), 87.

2 For the father of the concept ‘Living Law’, see: Eugen Ehrlich, Fundamental Principles of Sociology of Law (first published 1936, Piscataway, NJ: Transaction Publishers, 2009).

3 The Committee on Economic, Social and Cultural Rights (CtESCR), ‘General Comment No. 13 The Right to Education (Article 13 of the Covenant)’ UN doc. E/C.12/1999/10 (1999), para. 52; CtESCR, ‘General Comment No. 14 The Right to the Highest Attainable Standard of Health’ UN doc. E/C.12/2000/4 (2000), para. 52; CtESCR, ‘General Comment No. 15 The Right to Water’ UN doc. E/C.12/2002/11 (2002), para. 54; CtESCR, ‘General Comment No. 16 The Equal Right of Men and Women to the Enjoyment of All Economic, Social and Cultural Rights (art. 3)’ UN doc. E/C.12/2005/4 (2005), para. 39; CtESCR, ‘General Comment No. 17’ UN doc. E/C.12/GC/17 (2006), paras 49–50; and CtESCR, ‘General Comment No. 18: Article 6’, UN doc. E/C.12/GC/18 (2006), paras 46–7.

4 See for instance: Committee on the Rights of the Child (CtRC), ‘General Comment No. 5 General Measures of Implementation of the Convention on the Rights of the Child’, UN doc. CRC/GC/2003/5 (2003), paras 48–50; CtRC, ‘General Comment No. 7 Implementing Child Rights in Early Childhood’, UN doc. CRC/C/GC/7/Rev.1 (2006), para. 39; CtRC, ‘General Comment No. 9 The Rights of Children with Disabilities’, UN doc. CRC/C/GC/9 (2006), para. 18; CtRC, ‘General Comment No. 11 Indigenous Children and their Rights under the Convention’, UN doc. CRC/C/GC/11(2009), paras 26, 34 and 80; CtRC, ‘General Comment No. 13 The Right of the Child to Freedom From All Forms of Violence’, UN doc. CRC/C/GC/13 (2011); CtRC, ‘General Comment No. 15 on the Right of the Child to the Enjoyment of the Highest Attainable Standard of Health (art. 24)’, UN Doc. CRC/C/GC/15; and CtRC, ‘General Comment No. 17 on the Right of the Child to Rest, Leisure, Play, Recreational Activities, Cultural Life and the Arts (art. 31)’, UN doc. CRC/C/GC/17 (2013).

5 For a discussion on these and other standards see: Ann J. Rosga and Margaret L. Satterthwaie, ‘The Trust in Indicators: Measuring Human Rights’, Berkley Journal of International Law 27, no. 2 (2009): 23–7.

6 Hans Otto Sano and Hatla Thelle, ‘The Need for Evidence-Based Human Rights Research’, in Methods of Human Rights Research, ed. Fons Coomans, Fred Grünfeld, and Menno T. Kamminga (Cambridge: Intersentia, 2009), 96.

7 They define quantitative data as follows: ‘Data created and/or collected by researchers, primary evidence. This category includes a number of sub-categories such as surveys, e.g. quantitative household or individual data, perception surveys, i.e. surveys soliciting perceptions by the informants, expert based quantitative assessments, i.e. experts assessing e.g. numbers or levels of human rights violations, and quantitative organisations data, for instance lists or registrations of numbers of e.g. prisoners, collected by government or private organizations’ (Ibid., 96).

8 Qualitative data was defined as follows: ‘Evidence collected by researchers as part of qualitative interviews or of anthropological observation. Event based data: descriptive and quantitative recoding of events and situations often undertaken by human rights organizations in connection with atrocities and systematic or gross human rights violations’ (Ibid.).

9 Fons Coonmans, Fred Grünfeld, and Menno T. Kamminga, ‘Methods of Human Rights Research: A Primer’, Human Rights Quarterly 32, no. 1 (2010): 183.

10 See: 5. Indicators, questions and benchmarks.

11 For more on the obligations of results, see: Matthew C. R. Craven, The International Covenant on Economic, Social, and Cultural Rights: A Perspective on its Development (Oxford: Clarendon Press, 1995).

12 Hadi K. Lile, ‘FNs barnekonvensjon artikkel 29 (1) om formålet med opplæring: En rettssosiologisk studie om hva barn lærer om det samiske folk’ (PhD, Faculty of Law, University of Oslo, 2011).

13 Thomas Mathiesen, Retten i samfunnet: En innføring i rettssosiologi, 6th ed. (Oslo: Pax Forlag, 2011), 25.

14 Mathieu Deflem, Sociology of Law: Visions of a Scholarly Tradition (Cambridge: Cambridge University Press, 2008), 3.

15 Håkan Hyden, Rättssociologi som rättsvetenskap (Lund: Studentlitteratur, 2002).

16 Reza Banakar, ‘A Passage to “India”: Toward a Transformative Interdisciplinary Discourse on Law and Society’, Retfærd 92, no. 2001 (2001): 3; Reza Banakar ‘The Identity Crisis of a Stepchild – Reflections on the Paradigmatic Deficiencies of Sociology of Law’, Retfærd 81, no. 1998 (1998): 3.

17 Kristian Andenæs, Sosialomsorg i gode og onde dager (Oslo: Tano, 1992), 77; Hadi K. Lile, ‘FNs barnekonvensjon artikkel 29 (1) om formålet med opplæring: En rettssosiologisk studie om hva barn lærer om det samiske folk’ (PhD thesis, Faculty of Law, University of Oslo, 2011), 33–41.

18 Vilhelm Aubert, Torstein Eckhoff, and Knut Sveri, En lov i søkelyset: Sosialpsykologiske undersøkelse av den norske hushjelplov (Oslo: Universitetsforlaget, 1952).

19 Ibid., 5.

20 For example: Tone Sverdrup, ‘Mellom ekteskapskontrakt og lønnskontrakt’ (Report from Nordisk forskerkurs i kvinnerett, Skrift nr. 3, University of Oslo, 1982); Tone Sverdrup, ‘Lovvern for arbeidstagere i private hjem’ (Kvinnerettslig arbeidsnotat nr. 31, University of Oslo, 1984). Leif E. Moland, ‘Ingen grenser? Arbeidsmiljø og tjenesteorganisering i kommunene’ (Report 221, Fafo, 1997); Jon Anders Hasle, ‘Sist inn – først ut? Om etterlevelse av arbeidsmiljølovens oppsigelsesvernregler overfor innvandrere’ (Særavhandling, University of Oslo, 1995); Morten Kjelland, Hjemmearbeideres erstatningsrettslige vern (Oslo: Gyldendal Akademisk, 2002); Annika Pfannenstill, Rättssociologiska studier innom området autism: Rättsanvendingen i en kunskapskonkurrerande miljö (Lund Studies in Sociology of Law, 2002); and Marianne Smith and Marianne Glærum, ‘Avhendingsloven – til hjelp eller besvær? En studie av boligkjøperes kjennskap til avhendingsloven og deres adferd under kjøpeprosessen’ (Skriftserie nr. 60, Institutt for rettssosiologi, University of Oslo, 1998).

21 Aubert, Eckhoff, and Sveri, En lov i søkelyset, 51.

22 Ibid.

23 Ibid., 51 and 55.

24 Jørgen Dalberg-Larsen, Loven og livet: en retssociologisk grundbog (Copenhagen: Jurist- og Økonomiforbundets Forlag, 2005), 44.

25 This is my free translation from Norwegian: Gustav Haraldsen, Spørreskjemametodikk: etter kokebokmetoden (Oslo: Gyldendal, 1999), 24.

26 Sidsel M. Sverdrup, ‘Evaluering av lovers tilsiktede virkninger: En case-studie av markedsføringsloven’ (PhD, Faculty of Social Science, University of Oslo, 1997), 144–76.

27 Sidsel M. Sverdrup, Evaluering: Tilnærminger, modeller og eksempler (Oslo: Gyldendal, 2014), 98–115.

28 Ibid.

29 Ibid., 105–6.

30 Ibid., 103–5.

31 See: 1.2 Human rights focus.

32 For more on triangulation, see: Frieder Wolf, ‘Enlightened Eclecticism or Hazardous Hotchpotch? Mixed Methods and Triangulation Strategies in Comparative Public Policy Research’, Journal of Mixed Methods Research 4, no. 2 (2010): 144–67.

33 Michael Evans, ‘Reliability and Validity in Qualitative Research by Teacher Researchers’, in School-based Research: A Guide for Education Students, ed. Elaine Wilson (SAGE Publication, 2009), 120.

34 Todd Landman and Edzia Carvalho, Measuring Human Rights (Oxford: Routledge, 2010), 45–127.

35 Ibid., Ch. 6.

36 Ibid., 91.

37 Here are, for example, some references to studies done in Norway on legal aid: Ståle Eskeland and Just Finne, Rettshjelp (Oslo: Pax forlag, 1973); Ståle Ekseland and Just Finne, ‘Allmenhetens behov for rettshjelp’ (1973) Vol. 4 Jussens Venner; Jon T. Johnsen, ‘Rettshjelpbehovet – dets omfang og årsaker’ (Rapport fra rettshjelpseminaret, 1979); Jon T. Johnsen, Retten til juridisk bistand. En rettspolitisk studie (Oslo: Tano, 1987); Merethe S. Haugen and Elisabeth Vigerust, ‘Det udekkede behov for rettshjelp – et uløst problem’ (Særavhandling at the Law Faculty, University of Oslo, 1992); Hans Petter Graver, Ane Broch et al. ‘Rettshjelp 2001. Behovet for rettshjelp i Oslos befolkning – deriblant et utvalg innvandrerkvinner’ (Juss-Buss in cooperation with Unipub 2002).

38 Landman and Carvalho, Measuring Human Rights, 76–8.

39 David Cingranelli and David L. Richards, ‘Quantitative Studies’, in Encyclopedia of Human Rights, ed. David P. Forsythe (Oxford: Oxford University Press, 2009), 3.

40 Joseph Raz, The Authority of Law: Essays on Law and Morality (Oxford: Clarendon Press, 1979), Ch. 8.

41 Kevin E. Davis, B. Kingsbury, and S.E. Merry, ed., Governance by Indicators: Global Power through Classification and Rankings (Oxford: Oxford University Press, 2012), 9–10.

42 Terence C. Halliday, ‘Legal Yardsticks: International Financial Institutions as Diagnosticians and Designers of the Laws of Nations’, in Governance by Indicators: Global Power through Classification and Rankings, ed. Kevin E. Davis, B. Kingsbury, and S.E. Merry (Oxford: Oxford University Press, 2012), 180–1.

43 Ibid., 181.

44 They use the term ‘construct validity’, but it means the same thing as discussed above – legal relevance.

45 Malcolm Langford and Sakiko Fukuda-Parr, ‘The Turn to Metrics’, Nordic Journal of Human Right 30, no. 3 (2012): 222, 232.

46 Landman and Carvalho, Measuring Human Rights, 97–8.

47 See: 3.1 Evaluation models.

48 Landman and Carvalho, Measuring Human Rights, 9–31.

49 Todd Landman, Studying Human Rights (Oxford: Routledge, 2006); Landman and Carvalho, Measuring Human Rights, 80.

50 Eugene Ehrich, Fundamental Principles of the Sociology of Law, trans. W.L. Moss (Cambridge, MA: Harvard University Press, 1936), 493.

51 David Nelken, ‘Law in Action or Living Law? Back to the Beginning of Sociology of Law’, Legal Studies 4, no. 2 (1984): 157–74. See also: Håkan Hydén, Normvetenskap (Lund Studies in Sociology of Law, Lund University, 2002).

52 Gordon W. Allport, The Nature of Prejudice (first published by New York, NY: Perseus Books Publisher, 1988), cited in Charles Stangor, Stereotypes and Prejudice (Philadelphia: Psychology Press, 2000), 27.

53 Hans-Georg Gadamer, Truth and Method, trans. Joel Weinsheimer and Donald G. Marshall (first published 1960, New York: Continuum, 1998), 276.

54 Immanuel Kant, Groundworks of the Metaphysics of Morals, trans. and edited by Mary Gregor and Christine M. Korsgaard (first published 1785, Cambridge: Cambridge University Press 1997), 20.

55 There is absolutely no room in this article to dive into this comprehensive debate. For a good overview of the debate see: Raymond Wacks, Understanding Jurisprudence: An Introduction to Legal Theory, 3rd ed. (Oxford: Oxford University Press, 2005), 10–118.

56 The first to write about the State of Nature was: Thomas Hobbes, Leviathan (first published 1651, Penguin, 1985).

57 D. Dyzenhaus, ‘Positivism's Stagnant Research Programme’, Oxford Journal of Legal Studies 20, no. 4 (2000): 719: ‘Legal positivist today form by far the biggest camp within legal theory’. Steven Ratner and Anne-Marie Slaughter, ‘Appraising the Methods of International Law’, American Journal of International Law 93: 291, 293: ‘It remains the lingua franca of most international lawyers.’ Ronald Dworkin, Taking Rights Seriously, 2nd ed. (Cambridge, MA: Harvard University Press, 1978), vii; Jean d’Aspremont, Formalism and the Sources of International Law: A Theory of the Ascertainment of Legal Rights (Oxford: Oxford University Press, 2013), 20.

58 Maria Green, ‘What We Talk About When We Talk About Indicators: Current Approaches to Human Rights Measurement’, Human Rights Quarterly 23, no. 4 (2001): 1062, 1065.

59 The term human rights indicator is often used as a tool to measure if a human rights project (funded by some donor) has reached its aims.

60 For more on the concept of ‘legal relevance’ and validity, see section 6.

61 Malcolm Langford, ‘Social Rights Adjudication: An Interdisciplinary Perspective’ (PhD, Faculty of Law, University of Oslo, 2014); Martin Scheinin, ‘Economic and Social Rights as Legal Rights’, in Economic, Social and Cultural Rights: A Textbook, ed. Asbjørn Eide, Catarina Krause, and Allan Rosas, 2nd ed. (Leiden: Martinus Nijhoff, 2001), 29–54.

62 Paul Hunt, ‘Interim Report of the Special Rapporteur of the Commission on Human Rights on the Right of Everyone to Enjoy the Highest Attainable Standard of Physical and Mental Health’ (2003) A/58/427.

63 OHCHR, ‘Report on Indicators for Promoting and Monitoring the Implementation of Human Rights’ (2008) HRI/MC/2008/3; OHCHR, ‘Human Rights Indicators: A Guide to Measurement and Implementation’ (2012) HR/PUB/12/5.

64 Ibid.

65 Principles relating to the Status of National Institutions (The Paris Principles). Adopted by The General Assembly resolution 48/134 of 20 December 1993.

66 OHCHR, ‘Report on Indicators for Promoting and Monitoring the Implementation of Human Rights’ (2008) HRI/MC/2008/3, para. 19.

67 See above footnote 63.

68 Lile, ‘FNs barnekonvensjon artikkel 29 (1) om formålet med opplæring’, 71–3.

69 Erik A. Andersen and Hans-Otto Sano, Human Rights Indicators at Programme and Project Level: Guidelines for Defining Indicators, Monitoring and Evaluation (Danish Institute for Human Rights, 2006), 13–17.

70 See: 3.1 Evaluation models.

71 OHCHR, ‘Report on Indicators for Promoting and Monitoring the Implementation of Human Rights’ (2008) HRI/MC/2008/3; OHCHR, ‘Human Rights Indicators: A Guide to Measurement and Implementation’ (2012) HR/PUB/12/5.

72 Rachel Hodgkin and Peter Newell, Implementation Handbook for the Convention on the Rights of the Child (UNICEF, 2007).

73 Katarina Tomaševski, ‘Indikators’, in Economic, Social and Cultural Rights: A Textbook, ed. Asbjørn Eide, Catarina Kraus and Allan Rosas, 2nd ed. (Leiden: Martinus Nijhoff, 2001), 532.

74 Hodgkin and Newell, Implementation Handbook for the Convention on the Rights of the Child.

75 OHCHR, ‘Report on Indicators for Promoting and Monitoring the Implementation of Human Rights’ (2008) HRI/MC/2008/3; OHCHR, ‘Human Rights Indicators: A Guide to Measurement and Implementation’ (2012) HR/PUB/12/5.

76 See: 5. Indicators, questions and benchmarks.

77 Terence C. Halliday, ‘Legal Yardsticks: International Financial Institutions as Diagnosticians and Designers of the Laws of Nations’, in Governance by Indicators: Global Power through Classification and Rankings, ed. Kevin E. Davis, B. Kingsbury, and S.E. Merry (Oxford: Oxford University Press, 2012), 80–181.

78 Ottar Hellevik, Forskningsmetode i sosiologi og statsvitenskap (Oslo: Universitetsforlaget, 2009), 17.

79 Sverdrup, Evaluering, 105–6.

80 See footnotes 3 and 4.

81 CtESCR, ‘Canada: Sixth Periodic Report’ (23 April 2013) E/C.12/CAN/6.

82 Jon Christian Nordrum discusses some of these problems: Jon Christian Fløysvik Nordrum, ‘Evaluering av lover’, in Evaluering: Tradisjoner, praksis, mangfold, ed. Anne Halvorsen, Einar L. Madsen, and Nina Jentoft (Oslo: Fagbokforlaget, 2013), 994–5.

83 For more on minimum core rights, see for instance: Langford, ‘Social Rights Adjudication’, 132–4; and Katharine Young, ‘The Minimum Core of Economic and Social Rights: A Concept in Search of Content’, Yale Journal of International Law 33, no. 1 (2008): 113–75.

84 CtESCR, ‘General Comment No. 19 The Right to Social Security (art. 9)’, UN doc. E/C.12/GC/19 (2008), para. 76; CtESCR, ‘General Comment No. 18: Article 6’, UN doc. E/C.12/GC/18 (2006), para. 47. CtESCR, ‘General Comment No. 17’, UN doc. E/C.12/GC/17 (2006), paras 50. CtESCR, ‘General Comment No. 15 The Right to Water’, UN doc. E/C.12/2002/11 (2002), para. 54. CtESCR, ‘General Comment No. 14 The Right to the Highest Attainable Standard of Health’ UN doc. E/C.12/2000/4 (2000), para. 58.

85 Thomas Risse, Stephen C. Ropp, and Kathryn Sikkink, The Power of Human Rights International Norms and Domestic Change (Cambridge: Cambridge University Press, 1999).

86 This is based on an interpretation of CRC article 29 (1) (b) in relation to education about Sámi people in Norway. See: Lile, ‘FNs barnekonvensjon artikkel 29 (1) om formålet med opplæring’, 113–64.

87 For more on additive indexes and the Likert-scales, see: Hellevik, Forskningsmetode i sosiologi og statsvitenskap, 157–69.

88 For more on the measurement of attitudes: John B. McConahay, Betty B. Hardee, and Valerie Batts, ‘Has Racism Declined in America?: It Depends on Who is Asking and What is Asked’, Journal of Conflict Resolution 25 (1981): 568.

89 See also: 4.1 What is human rights law?

90 Having said that, statements from the committees can develop international customary law and be an expression of that. If one can refer to a range of statements about the same thing many times, without any objections from states, one might argue that this interpretation from the committee is legally binding.

91 See: 5.4 Benchmarks.

92 Landman, Studying Human Rights; Landman and Carvalho, Measuring Human Rights, 76–8.

93 See: 4.1 What is human rights law?

94 Langford, ‘Social Rights Adjudication’, 128–32; Scheinin, ‘Economic and Social Rights as Legal Rights’, in Economic, Social and Cultural Rights, 29–54.

95 See: 5.5 ‘Should’ and ‘must’ assessments.

96 See: 5.3 Survey questions.

97 Ibid.

98 See: 5.4 Benchmarks.

99 Sverdrup, Evaluering, 103–5.

100 Landman and Carvalho, Measuring Human Rights, 45–127.

101 Landman, Studying Human Rights; Landman and Carvalho, Measuring Human Rights, 80.