Philosophical ruminations on measurement: methodological orientations of patient reported outcome measures (PROMS)

In many areas of medicine, including mental health, patients’ views of their own health and well-being are considered essential for improving the quality and cost effectiveness of health care. This has resulted in the development of myriad questionnaires, rating scales and assessment forms known as patient reported outcome measures (or PROMs) (Dawson, 2009; Trujols et al., 2013). PROMs seek to measure health status or health-related quality of life from the patient’s, rather than the clinician’s, perspective. They can be used in clinical practice to monitor patient progress, public health research to gauge treatment needs and clinical trials to evaluate outcomes following psychosocial interventions or medical procedures. PROMs can be generic (e.g. the Short-Form 36 [SF-36] which assesses self-perceived health via 36 questions relating to 8 broad domains of physical and psychological health) or condition specific (e.g. the Beck Depression Inventory [BDI] which measures the severity of depression via 21 questions).

PROM development is a sophisticated process, the methodology of which is constantly evolving. The individual questions (or “items”) that comprise PROMs tend to be selected using a variety of techniques, including reviews of previous questionnaires and the literature, clinical observations, theory and qualitative methods. Once developed, PROMs are subject to rigorous psychometric testing to ensure that they are reliable, valid, and measure health status and health care outcomes as objectively as possible. The involvement of patients in PROM development is deemed essential (Patrick et al., 2008), and helps to ensure that the constructs measured, the questions included and the language and terminology used are acceptable to, and reflect the priorities and preferences of, the target patient populations. Within the mental health field, PROMs have also been developed using innovative participatory methodology (Evans et al., 2012; Rose et al., 2011; Trivedi & Wykes, 2002). For example, Views on Inpatient Care (VOICE) is a 19-item measure that was designed with significant input from mental health service users, including people who had been detained under the 1983 Mental Health Act and service user researchers (Evans et al., 2012).

Despite an extensive and expanding literature on PROMs, little has been written on the philosophical and methodological orientations of these measures. This seems surprising given that their development and subsequent validation involves a complex mixture of qualitative and quantitative methods, subjective and objective measures, and personal and clinical perspectives. PROMs, particularly those involving early and meaningful engagement with purposively-selected groups of patients and service users, can certainly enhance our understanding of how people feel about their health and consequently how best to measure self-perceived health status. However, they also have limitations given that their ultimate goal is to translate complex personal feelings and experiences into simple numeric scores. Thus, PROMs tend to gloss over human diversity and the day-to-day, moment-by-moment, fluctuations in how people perceive their health and health status. Instead, they commonly assume that people have similar fixed needs and preferences, and will all attribute the same significance to PROM items and constructs. This quasi “objectivity” creates tensions for qualitative researchers whose more interpretative approaches uphold subjectivity, context, meaning and change.

Some simple examples from addiction, which is our own area of research and practice, can help to illustrate these problems. Abstinence from illicit drugs, teetotalism and smoking cessation may all “objectively” appear to be good health outcomes. However, we cannot assume that these are desired goals for all people at all times. Some individuals enjoy taking illicit drugs, some feel that drinking alcohol enhances their sociability, and others believe that tobacco smoking brings relief from anxiety. In order to gauge the actual significance of abstinence as a measure of treatment success for any given individual, we also need to know how much value a person puts on this achievement relative to other substance-related outcomes (such as reduced illicit drug use, moderate drinking or moving from smoking cigarettes to nicotine replacement therapy) and to their broader life goals (such as having stable relationships, securing a job or saving money). Individuals are often more concerned about immediate negative social and interpersonal outcomes than long-term health problems. This has long been recognized in public health campaigns where the risk of spots, erectile dysfunction and bad breath (rather than more life-threatening morbidities and mortality) have been emphasized in order to deter drug use, excessive alcohol consumption and smoking, respectively.

In recent years, newer forms of preference-based and individualized PROMs have sought to better capture the relative importance of particular outcomes to patients (Bridges et al., 2007; Garratt et al., 2002). While these measures permit greater personalization of scoring systems, they are still inherently reductionist – condensing complex, fluctuating human feelings and experiences into a series of standardized digits, and so denying the patient or service user the opportunity to elaborate on how and why they feel the way they do and how and why that might be qualitatively different from the way they felt yesterday, 1 week ago or 1 year ago. Similarly, patient preference-based measures – like other PROMs – tend to assume that all the individuals share a basic common understanding of the constructs and the attributes being measured. Yet this is not necessarily the case. Turning to the addiction field once more, constructs such as “cravings”, “dependence” or “recovery” can all mean different things to different people. If we cannot define a construct or its constituent elements with precision or in a way that most people understand and agree on, how can we hope to measure it?

In promoting the need for good qualitative research methods in developing PROMs with content validity, it has been argued that phenomenology – with its emphasis on understanding the lived experience of individuals – is the sine qua non overarching framework for instrument developers (Lasch et al., 2010). Certainly, phenomenology provides a neat counter to the concerns of reductionism we have identified above. Yet we do not believe that it provides a satisfactory solution. This is because philosophical and methodological calls to individual experience, when taken to their logical conclusion, end in the rather thorny terrain of post-structuralism. Here, there are no common understandings or shared experiences at all. Instead, difference and diversity dominate, language and representation are ranked above materialism, and the meaning of all concepts, including those relating to health status and health-related quality of life, are dismissed as fundamentally unstable. Post-structuralism is in essence antithetical to measurement, negating the many experiences that people share and denying the ways in which lives are physically and materially constrained by health problems. PROMs must certainly capture patients’ perspectives in order to be valid, but they also need to measure something more concrete than discourse and representation if they are to be reliable or useful.

In terms of the philosophical paradigm of the PROM, this creates an obvious challenge. PROMs need to reflect individual experiences (interpretivism) but ultimately yield a numeric score on a scale that represents a pre-defined construct (positivism). We hypothesize that PROMs are perhaps best viewed within the philosophical tradition of pragmatism. This is because they are likely to have most benefit when we eschew endless debates about methodology and the meaning of knowledge, concepts or reality, and instead focus on their practical application, utilizing the most appropriate methods to facilitate their development. Well-designed PROMs – based on sound qualitative methods followed by rigorous psychometric testing – can enable patients to rapidly convey what they are feeling and experiencing in ways that researchers and clinicians can understand, harness and use to improve treatment. Nonetheless, the utility of any quantitative measure they produce is likely to be enhanced if patients are able to supplement their numeric scores with more in-depth commentary on why they feel the way they do and whether and how some elements of a scale have more or less relevance to them than others at particular moments in time.

Eliciting these more personalized and fluid accounts of health status and health-related quality of life requires further consideration by PROM developers. Our recent experiences of developing a PROM for addiction recovery, with significant service user involvement and combining qualitative and quantitative methods, have been illuminating in this respect (Neale et al., 2014, 2015). First, we found that service users routinely disagreed on many aspects of the PROM content, particularly the questions to be included and terminology to be used. Despite this, they were consistently tolerant of each other’s opinions – routinely arguing that “it will be different for others” and “I can only tell you from my experience”. Additionally, and in spite of their many differences of opinion, it was still possible to identify a basic bank of questions and forms of phrasing that nearly all the service users deemed acceptable. From this, we began to appreciate that we could never develop a “perfect” PROM for recovery but also that service users were not expecting that. What we were starting to produce was a set of questions that made the complex and contested concept of recovery interpretable and tangible; resulting in a PROM that had the characteristics of a “working consensus” rather than a measure of truth (Habermas, 1970).

Secondly, an interesting unplanned development was that, as our PROM developed and piloting and psychometric testing began, service users repeatedly told us that they “enjoyed completing the questionnaire”, often adding “because it made me think about areas of my life I need to think about”. In fact, completion of the PROM often took longer than anticipated because service users routinely stopped to reflect on, and discuss at length, items that they felt were particularly important to them personally. From this, it became apparent that completion of our PROM seldom simply generated a numeric score. Rather it was eliciting detailed contextual information, functioning as a source of self-reflexivity as individuals considered how their responses today compared with how they might have responded in the past or in the future. In short, the PROM was indirectly producing rich qualitative data that could form the basis of subsequent therapeutic work within the clinical setting, while also having potential value for more interpretive qualitative analyses. These unexpected methodological observations seem to support our suggestion that PROMs are best viewed through the lens of pragmatism. They cannot, and should not, promise an objective “truth” about self-perceived health status. Nonetheless, they offer many valuable uses, particularly if they are designed using an appropriate combination of methods that embraces a multi-perspective paradigm (Rose et al., 2006).

Declaration of interest

JN is part-funded by, and John Strang is supported by, the National Institute for Health Research (NIHR) Biomedical Research Centre for Mental Health at South London and Maudsley NHS Foundation Trust and King's College London. John Strang is a clinician and has had, and continues to have, clinical responsibilities and has also worked with a range of types of treatment and rehabilitation service-providers. He has also worked with pharmaceutical companies to seek to identify new or improved treatments, and also with a range of governmental and non-governmental organisations. A fuller account is given on his personal university webpage at: http://www.kcl.ac.uk/ioppn/depts/addictions/people/hod.aspx. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.