Exploring how to evaluate a qualitative patient-centered outcome measure: literature review and illustrative example – a Perthes child-friendly measure

Figures & data

Table

A Muse: I wonder how we might evaluate a measure which comprises a majority of open-ended questions, including perhaps use of emojis, story writing and pictures. How best, and in a credible and authoritative way, can this be done and thus demonstrate its validity, reliability and responsiveness to change?

Table 1 Overview of databases, search terms and yield

Database	Search terms	Yield (number of papers)
PUBMED (MESH Terms)	Psychometrics; outcome assessment; health care	162
	Outcome assessment; health care; psychometrics; qualitative research	146
	Patient reported outcome measures; outcome assessment; health care; methods. Subheading: methods.	155
SCOPUS (Word Search Terms)	Qualitative, outcome measure, scale, development, validation	125
	Qualitative outcome measure, scale development. Subheading: psychometrics	88
Web of Science (Word Search Terms)	Qualitative research, outcome measure, psychometric validation	236

Table 2 Illustrative examples of group two articles by theme

Thematic area	Illustrative articles and content
Methodologically/Method Oriented	A: Discussion of “What is Validity and Reliability” in Qualitative Research: Winter^Citation15: for example, points to. a “realist” approach – an account is valid if it reflects the perspectives of the actors in that situation (p. 7) Creswell & Miller^Citation16: stress importance of exploring validity from the lens of participants, for example, via member checking and peer (and cognitive) debriefing Golafshani^Citation17: inter alia, argues (p601) that validity, and reliability, in qualitative paradigms are assessable in terms of Credibility, Neutrality or Confirmability, Consistency or Dependability and Applicability or Transferability^Citation18 and of the importance of triangulation from multiple perspectives Frost et al^Citation19: focus on what are the psychometric properties to generate “sufficient evidence” for the validity and reliability of a PROM; argues for importance of establishing content validity as primary task, and use of qualitative research to do this, in particular, focus group and cognitive interviews; and then use of psychometric validation approaches
Theoretical Discussions of Validity	B. Theoretical/Philosophical Discussions Zumbo^Citation20: what is validity, and particularly construct validity, and its implications for process of validation? Presents a “contextualised and pragmatic explanation” of validity; construct validity ‘‘should provide an explanation for the test scores for the observed variation in test scores’’ (p. 69); validity as “establish (ing) the “why” and “how…” (p. 70) and as “support (ing) inferences … (made) from test scores…” (p. 70); validation is a “higher order integrative process…involving…concept formation….” (p. 69); argues for “multilevel testing and measurement” for a multilevel construct (p. 78). Gadermann et al^Citation21: importance of asking, “what are the underlying cognitive processes that result in respondents providing responses to self-report questions” (p. 39) in the way they do; use cognitive interviewing to do this. Hubley & Zumbo^Citation22: explore meaning of 'response processes'; argue that response processes should be considered as “mechanisms that underlie what people, do, think or feel...when...responding to an item” (p. 2); research in this area should “become more explanation-based” (p. 8) and explore “the broader context (i.e. purpose of testing, setting, culture)” (p. 8) when the response to an item is completed.
Guides to Best Practice for Measure Development	Wild et al^Citation23: exploring best practice for cultural adaptation and translation of a PROM; includes use of persons in new cultural context with experience in qualitative interviewing and/or cognitive interviewing to explore translation and cultural adaptation Brod et al^Citation24 present best practice guide in use of qualitative research and exploration of content validity Luyt^Citation25: drawing on Adcock and Collier^Citation26 presents a framework for measure development comprising three inter-connected stages: (i) measure development (background concept; developing concept definition; devising indicators); (ii) measure validation and (iii) measure revision. Advocating use of qualitative (in stage 1) and quantitative (for stage 2).
Use and Importance of Qualitative Research in Development of a PROM	Lasch et al^Citation27: qualitative research as providing sound and rigorous basis for PROM development; role of theoretical saturation (in coding categories) and triangulation, to explore from multiple perspectives Cheung and Clark^Citation28: major role of qualitative research in PROM development and also cultural adaptation
Use of Qualitative Approaches in Constructing a Measure and Generating the Item Pool	Mallinson^Citation29: importance of exploring the meaning and interpretation that respondents place on items in a PROM, focusing here on the SF-36, a fixed-choice measure; item interpretation and meanings attached may interact with a range of social and cultural factors affecting the respondent; use of face-to-face (cognitive, debriefing) interviews while respondent completes the measure Viswanathan et al^Citation30: explore measurement implications of scale responses, depending on whether the primary concern is maximizing discrimination between scale responses, whilst retaining reliability) and meaningful discrimination from the perspective of the respondent; argues for greater emphasis to be placed on meaningful discrimination from the perspective of the potential respondent, and not measure developer/researcher. Luyt^Citation25: explores measure development phase, in particular, the “constellation of meanings and understandings associated with a given concept” (p. 4), using focus groups Cheung and Clark^Citation28: significance in ensuring explicit focus on patient perspectives; critical role lies in both item generation and establishing content validity (eg, concept elicitation, cognitive debriefing) Breyer et al^Citation31: develop a patient-grounded measure on the symptoms, functions and impacts of urethral stricture disease; use of concept elicitation and cognitive interviews, followed by patients prioritizing items in terms of their bothersome-ness
Use of Qualitative Approaches in Establishing Construct Validity and, In particular, Content Validity of a Patient Reported Outcome Measure (PROM)	Hardesty and Bearden^Citation32: explore the use of expert judges in assessing the face and content validity of items Cremenns et al^Citation33: present a literature review of health self-report measures for children aged 3–8 years; range of measures found, using formats of Likert scales, graphical (pictorial), facial (cartoon) or visual analog; in 40% of measures children involved in item development (researcher talking with child) and in 47% pilot testing with children, where authors reported on content validity, in 40% of children themselves informed this; argues that measure developers should draw on the child’s perspective from the child, and not just rely on researcher/expert panel
Use of Qualitative Approach to Explore the Validity or Reliability of a Qualitative Measure	Golafshani^Citation17: advocates the use of quality criteria drawn from Lincoln & Guba^Citation18 – credibility, confirmability, consistency transferability; and exploring from multiple perspectives Cremenns et al^Citation34 : explore the development of a generic quality of life measure for school-age (6–9 year old) children; advocate and use “think aloud”/cognitive interviews; develop coding categories to select 30 items for the measure; and strategy used by children to answer items for the measure.; use of two independent raters, leading to exploration of intra- and inter-rater reliability; use of both qualitative measures (measure development) and quantitative (for reliability testing and comparison of strategy use) Gadermann et al^Citation21: develop coding and sub-coding categories, guided by research purposes (in the paper, strategies employed by children to respond to measure’s items); present in tree diagram format; illustrate category content; add frequency counts; compare tree diagrams with counts attached (here by strategy categories in item response – absolute, relative, general positive, unclear) Luyt^Citation25: suggests the use of multiple (two or more) coders for qualitative data analysis, then exploring of inter- and intra-rater reliability quantitatively

Box S1 Opening page of measures.

Box S2 Example of items to rate with an Emoji.

Box S3 Story writing.

Winter G. A comparative discussion of the notion of ‘validity’ in qualitative and quantitative research. Qual Rep. 2000;4(3/4):1–14.

 Google Scholar

Creswell JW, Miller DL. Determining validity in qualitative inquiry. Theory Pract. 2000;39(3):124–131. doi:10.1207/s15430421tip3903_2

 Web of Science ®Google Scholar

Golafshani N. Understanding reliability and validity in qualitative research. Qual Rep. 2003;8(4):597–607.

 Google Scholar

Lincoln YS, Guba EG. Naturalistic Inquiry. Beverly Hills, CA: Sage; 1985.

 Google Scholar

Frost MH, Reeve BB, Liepa AM, Stauffer JW, Hays RD. The Mayo/FDA patient-reported outcomes consensus meeting group. What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value Health. 2007;10(Suppl 2):S94–S105. doi:10.1111/j.1524-4733.2007.00272.x17995479

 PubMed Web of Science ®Google Scholar

Zumbo BD. Validity as contextualized and pragmatic explanation, and its implications for validation practice In: Lissetz RW, editor. The Concept of Validity: Revisions, New Directions and Applications. Charlotte: Information Age Publishing, Inc.; 2009:65–82.

 Google Scholar

Gadermann AM, Guhn M, Zumbo BD. Investigating the substantive aspect of construct validity for the satisfaction with life scale adapted for children: a focus on cognitive processes. Soc Indic Res. 2011;100(1):37–60. doi:10.1007/s11205-010-9603-x

 Web of Science ®Google Scholar

Hubley AM, Zumbo BD. Response processes in the context of validity: setting the scene. In Zumbo BD, Hubley AM, editors. Understanding and Investigating Response Processes in Validation Research Social Indicators Research Series 69, Springer International Publishing AG; 2017:1-12.

 Google Scholar

Wild D, Grove A, Martin M, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR task force for translation and cultural adaptation. Value Health. 2005;8(2):94–104. doi:10.1111/j.1524-4733.2005.04054.x15804318

 PubMed Web of Science ®Google Scholar

Brod M, Tesler LE, Christensen TL. Qualitative research and content validity: developing best practices based on science and experience. Qual Life Res. 2009;18(9):1263–1278. doi:10.1007/s11136-009-9540-919784865

 PubMed Web of Science ®Google Scholar

Luyt R. A framework for mixing methods in quantitative measurement development, validation, and revision: a case study. J Mix Methods Res. 2011;6(4):294–316. doi:10.1177/1558689811427912

 Web of Science ®Google Scholar

Adcock R, Collier D. Measurement validity: a shared standard for qualitative and quantitative research. Am Polit Sci Rev. 2001;95:529–546. doi:10.1017/S0003055401003100

 Web of Science ®Google Scholar

Lasch KE, Vigneux M, Abetz L, et al. PRO development: rigorous qualitative research as the crucial foundation. Qual Life Res. 2010;19:1087–1096. doi:10.1007/s11136-010-9655-z20512662

 PubMed Web of Science ®Google Scholar

Cheung KKF, Clark AM. Qualitative methods and patient-reported outcomes: measures development and adaptation. Int J Qual Methods. 2017;16:1–3.

 Web of Science ®Google Scholar

Mallinson S. Listening to respondents: a qualitative assessment of the short-form 36 health status questionnaire. Soc Sci Med. 2002;54:11–21. doi:10.1016/s0277-9536(01)00003-x11820675

 PubMed Web of Science ®Google Scholar

Viswanathan M, Sudman S, Johnson M. Maximum versus meaningful discrimination in scale response. Implications for validity of measurement of consumer perceptions about products. J Bus Res. 2004;57:108–124. doi:10.1016/S0148-2963(01)00296-X

 Web of Science ®Google Scholar

Breyer BN, Edwards TC, Patrick DL, Voelzke BB. Comprehensive qualitative assessment of urethral stricture disease: toward the development of a patient centered outcome. J Urol. 2007;198:1113–1118. doi:10.1016/j.juro.2017.05.077

 Web of Science ®Google Scholar

Hardesty DM, Bearden WO. The use of expert judges in scale development: implications for improving face validity of measures of unobservable constructs. J Bus Res. 2004;57:98–107. doi:10.1016/S0148-2963(01)00295-8

 Web of Science ®Google Scholar

Cremeens J, Eiser C, Blades M. Characteristics of health-related self-report measures for children aged three to eight years: a review of the literature. Qual Life Res. 2006;15(4):739–754. doi:10.1007/s11136-005-4184-x16688506

 PubMed Web of Science ®Google Scholar

Cremeens J, Eiser C, Blades M. A qualitative investigation of school-aged children’s answers to items from a generic quality of life measure. Child Care Health Dev. 2007;33(1):83–89. doi:10.1111/j.1365-2214.2006.00665.x17181757

 PubMed Web of Science ®Google Scholar