Patient-reported outcomes validated in phase 3 clinical trials: a targeted literature review

Abstract

Background

Regulatory guidance advises validation of patient-reported outcome (PRO) instruments prior to use in pivotal clinical studies, which may then be used to generate critical patient-centered evidence and support labelling claims. This targeted literature review aimed to determine if PRO instruments psychometrically validated in a phase 3 trial setting could support label claims from the same phase 3 study (i.e. PRO data were generated as an endpoint).

Methods

A targeted search of published studies (1 January 2006−3 June 2021) using the MEDLINE database identified PRO instruments validated during phase 3 trials. The search included instrument terms (e.g. patient-reported outcome measures, questionnaire, survey) and validation terms (e.g. reproducibility, minimal important difference), without filtering for therapeutic indications. Results were limited to phase 3 clinical trials or validation studies. The PROLABELS database was used to identify PROs validated in phase 3 trials and accepted in labelling claims.

Results

Of 355 references identified, 68 studies with PRO psychometric validation in phase 3 studies were selected, covering 78 instruments. Of these, 20 were novel PRO instruments and 58 were existing instruments being validated for a new therapeutic indication/population. The psychometric properties most frequently validated were internal consistency reliability, known-group validity, responsiveness, minimal important difference, and concurrent validity. Five novel instruments obtained ten labelling claims for seven drugs/products.

Conclusions

These results suggest that quantitative validation of novel PRO instruments, and existing PROs for new indications, can occur within the context of phase 3 trials, and these PROs can also support label claims.

Keywords:

• Targeted literature review
• patient-reported outcome
• phase 3 trial
• validated

Introduction

A patient-reported outcome (PRO) is a health-related assessment that is based solely on the patient’s perspective, without interpretation by a clinician or others^1,2. PROs can be used as primary or secondary endpoints in clinical trials, or as a tool for patient-centered care in clinical practice^3–5. When incorporated into clinical trials, PROs can provide an important assessment of the patient’s experience of a disease and the overall impact of a health intervention, giving a perspective on how patients feel and function that may not be adequately captured by other clinical endpoints^2,6.

The information gained from PROs may include a measure of symptoms (e.g. pain), well-being, medication side-effects, satisfaction with care, functional status (physical, sexual, social, emotional, or cognitive), or multidimensional constructs such as health-related quality of life (HRQoL) or health utility^2,5,7. PRO instruments are increasingly being recognized by regulators as important tools to collect patient-centered evidence^1,8,9. However, it is critical that properly validated instruments are used to ensure standardized assessment so that any differences in patient responses are based on robust and clinically meaningful differences in patients’ experiences, instead of variations in study design or biases⁵. One such example can be seen with time-sensitive experiences such as chemotherapy trials, where PRO assessments should capture the time periods in which patients experience the effects of chemotherapy; if discounted from study design, when PRO assessments are conducted can lead to biases⁵. PRO instruments are generally developed with input from various stakeholders, including clinicians, patients, and psychometric experts, to determine the most clinically relevant PRO domains and conceptual framework^5,10, and to identify an approach to measure these domains, which maximizes the relevance of each question^2,5.

While appropriate use of PRO data can support medical labelling claims, clinical trials that have not selected an appropriate PRO measure, or one that has not been properly validated, may have subsequent PRO-based labelling claims declined^5,11. The US Food and Drug Administration (FDA) specifies that PRO data can support labelling claims if the PRO instrument is well-defined, reliable, used in accordance with the instrument’s documented measurement capability, and validated in the target population¹. Guidance from Europe also indicates that PRO validation should preferably take place ahead of their inclusion in a phase 3 confirmatory trial⁸. However, early PRO instrument validation can be challenging since the outcomes and measures that are most important for the target population are often selected late in the clinical trial program¹². For this reason, PRO analyses tend to be more prominent in post-marketing, observational research, rather than pre-approval clinical trial programs¹², limiting their inclusion in labelling claims and availability of patient-reported information to healthcare providers, patients, and other stakeholders. Omission of PRO data creates an incomplete picture, particularly for patients deciding whether to start treatment, for whom information about the user experience is particularly meaningful.

Understanding how PRO instruments have been validated and used to generate evidence in the phase 3 study setting will help inform the design of future trials with a view to greater PRO inclusion. Similarly, analysis of the PRO instruments that have been accepted in medical labelling claims can also inform the future trial design. To investigate these factors, in this paper we report findings from a targeted literature review that aimed to identify novel PRO instruments (or existing instruments for new indications) that were quantitatively validated in a phase 3 setting and accepted in any labelling claims. Validated PRO instruments identified during the targeted literature review were then reviewed to determine if any had been accepted in labelling claims using the data from the same phase 3 study (i.e. PRO data were generated as an endpoint) and/or subsequent trials or additional evidence.

Material and methods

Study design

A two-stage targeted literature review was conducted. This type of review (also known as a focused literature review) is appropriate when the review is limited to describing the implications of a single, specific aspect of previous research, such as the methodology¹³. As this approach focuses on a single clearly formulated question, the approach is less rigorous than a systematic literature review and is typically performed by a single reviewer using explicit criteria for identification, selection, and critical analysis of a particular element of past research.

Stage 1 consisted of a literature review of published quantitative studies which aimed to identify PRO instruments validated during phase 3 trials and to provide information on the therapeutic area and outcomes of the validation. During stage 2, a review was undertaken to identify which of the PROs validated in phase 3 trials had been accepted in labelling claims using the same phase 3 study and/or which PROs had supported labelling claims based on other trials or later sources of evidence.

Stage 1

A targeted literature search of the MEDLINE database (via the Ovid platform¹⁴) was used to identify phase 3 studies with validated PRO instruments. The search strategy included instrument and validation terms, without filtering for therapeutic indications (Table 1). The search criteria were limited to phase 3 clinical trials or validation studies from 1 January 2006 to 3 June 2021. Retrieved references were reviewed carefully. The selection and data extraction processes were monitored in three steps:

Table 1. MEDLINE database search strategy.

Search #	Algorithm
1. Instrument terms	Exp Surveys and Questionnaires/ OR Exp Self-report/ OR Exp Patient reported outcome measures OR Exp patient outcome assessment/ OR (Questionnaire^a or Scale^a or Instrument^a or diaries or diary or Survey^a or patient-report^a or self-administ^a or self-report^a or Tool^a or Index or Inventory or checklist^a or PROMS).ab,ti.
2. Validation terms	exp Psychometrics/ OR exp Validation study/ OR exp validation studies as topic/ OR exp Reproducibility of results/ OR (Validation or validity or responsiveness or ability to detect change or minim^a important difference^a or MID or minim^a clinical^a important difference^a or MCID or reliability or internal consistency or Cronbach or know-group^a or test-retest or inter-rater or inter-observer or intra-rater or intra-observer).ab,ti.
3. Study terms	(Phase III or phase 3).ab,ti.
Limits	Abstract, English language, Human, Last 15 years, Publication type: Clinical trial, phase III or Validation study.
Final algorithm	#1 AND #2 AND #3 AND Limits

Abbreviations. MCID, minimal clinically important difference; MID, minimal important difference; PROMS, patient-reported outcome measures.

Exp Term/, search in Mesh-Tree exploded; ab.ti., search performed in title and abstract field.

^aUnlimited truncation, any word that begins with the term will be retrieved.

• Step 1: titles/abstracts were reviewed according to selection criteria for full-text review.

• Step 2: full-text articles were reviewed according to selection criteria for data extraction.

• Step 3: relevant data were extracted from selected articles.

Titles and abstracts were screened using inclusion criteria of the selection process and to remove duplicates. Inclusion required the study design to be a phase 3 clinical study, a population of all therapeutic indications and an outcome of PRO instruments validation.

Relevant data were extracted for all the PRO validation studies including the publication reference, study location, objective, trial objective, sponsor, therapeutic area, study population (sample size, age, gender), study type (e.g. observational, placebo-controlled), intervention/comparator, PRO validating steps taken in the context of the phase 3 trial (psychometric properties assessed, minimal important differences derived, anchor used) and outcome of the validation (i.e. Cronbach’s alpha for internal consistency reliability).

Trial designs were examined using ClinicalTrials.gov and crosschecked to ensure that the named clinical trials included the PRO instrument under validation as an outcome measure. The trial name or the NCT number was searched in CT.gov, using the “Other terms” or “Title/Acronym” sections. Once the trial was retrieved, the “Outcome Measures” section was screened for the presence of the PRO and the initial version of the trial was compared to the last one using the “Study Record Versions” tool of the database. The screening of the references from Medline and data extraction were performed by a single author and reviewed by an additional author. Reference selection was agreed upon by all four authors.

Stage 2

The PROLABELS database (accessed via the ePROVIDE platform at https://eprovide.mapi-trust.org/ – a database on Clinical Outcome Assessment (COA) endpoint strategies that are used to demonstrate the efficacy and safety of drugs or devices in the market approval process) was used to identify the pharmaceutical products including PROs validated in phase 3 trials and with accepted labelling claims. The database was searched by two authors for label claims up to and including 8 September 2021 and cross-checked for accuracy.

A search was performed under the advanced search engine of the PROLABELS database using the following selection criteria: Instruments - name of instrument selected; and Drug status: Approved or Withdrawn. The identified products, corresponding labels, and Summary of Product Characteristics (SmPC), were reviewed by two authors to ensure accuracy, and the following information was extracted: listing of the international non-proprietary name (INN); brand name; reference number; regulatory agency (FDA, European Medicines Agency (EMA)); marketing authorization holder; product approval date; specific therapeutic indication; study design; claim approval/revision date(s); PRO evaluations endpoint positioning; outcome assessment; name of PRO instrument and PRO language (PRO results) in the label; URL link to the label/SmPC.

Results

Targeted literature review (Stage 1)

Overall, 355 publications that included PROs were identified. After screening for relevant data, 68 studies with PRO psychometric validation were selected for in-depth review (Figure 1, supplemental Table S1). Of the 287 excluded studies, the primary reason for study exclusion was lack of validation (103, 35.9%; Figure 1). The 68 studies included in the analysis covered 78 unique PRO instruments that were validated within phase 3 clinical trials. Of these, 20 (25.6%) were novel PRO instruments and 58 (74.4%) were pre-existing instruments being validated for a therapeutic indication/population different from that for which it was originally developed (supplemental Table S2).

Figure 1. PRISMA flow diagram for the selection process of the references retrieved by the MEDLINE search. Abbreviations. COA, clinical outcome assessments; PRO, patient reported outcomes

Clinical study review and labelling claims (Stage 2)

Labelling claims

A review of the PROLABELS database revealed that five novel PRO instruments (Table 2) supported 10 labelling claims for seven different products (five from EMA; two from FDA) (Figure 2). The Oral Mucositis Daily Questionnaire (OMDQ) and Varicose Veins Symptom Questionnaire (VVSymQ) instruments obtained a claim using the same study as the psychometric validation. PRO endpoints were used as primary endpoints in four instances, as secondary endpoints in four instances, and were not specified in two instances.

Figure 2. Labelling claims flow diagram. ^aData on study design, therapeutic indication, and psychometric assessment results extracted from 21 studies for all the 20 instruments validated in the publications. Abbreviations. OMDQ, Oral Mucositis Daily Questionnaire; PEP, Premature Ejaculation Profile; PDQ, Peyronie’s Disease Questionnaire; PRO, patient reported outcome. WPS-RA, Rheumatoid arthritis-specific Work Productivity Survey; VVSymQ, Varicose Veins Symptoms Questionnaire.

Table 2. PROs with label claim.

PRO instrument	Drug brand name	Status	Regulatory agency	Therapeutic indication	Study summary	Claim approval date	Concept of interest	Endpoint	Claim
OMDQ Oral Mucositis Daily Questionnaire	Kepivance	Withdrawn	EMA	Mucositis	Study 2	10/25/2005	Mouth and throat soreness	Secondary	Kepivance [Palifermin] treated patients demonstrated significant benefits in patient-reported outcomes of mouth and throat soreness and its impact on swallowing, drinking, eating, and talking
PDQ Peyronie’s Disease Questionnaire	Xiaflex	Approved	FDA	Penile induration	Studies 1 and 2	12/06/2013	Symptom bother	Primary	Xiaflex significantly reduced patient-reported bother associated with Peyronie’s disease compared with placebo. The reduction in the bother domain score was numerically similar between patient groups stratified by degree of baseline curvature deformity (30–60 degrees and 61–90 degrees).
	Xiapex	Withdrawn	EMA	Penile induration	AUX-CC-803 and AUX-CC-804	02/02/2015	Erectile function	Primary	Xiapex significantly reduced patient-reported bother associated with Peyronie’s disease compared with placebo. The reduction in the bother domain score was numerically similar between patient groups stratified by degree of baseline curvature deformity (30–60 degrees and 61–90 degrees). Through the 5-year follow-up period, subjects demonstrated an improvement in penile curvature and in PDQ bother compared with the last observed value from the previous phase 3 studies.
					Study AUX-CC-806	05/26/2016	Erectile function	Primary
					Long-term follow-up study AUX-CC-810	05/29/2019	Erectile function	Not specified
PEP Premature Ejaculation Profile	Fortacin	Approved	EMA	Premature ejaculation	Study 1	11/15/2013	Ejaculatory control and difficulty, sexual satisfaction, distress	Secondary	The PEP scores followed a similar pattern of improvement to the IELT and IPE scores. For all of the three-monthly assessments completed by the subjects, there was a significant difference between Fortacin/Senstend and placebo (p < .0001).
	Senstend	Approved	EMA	Premature ejaculation	Study 1	11/14/2019	Ejaculatory control and difficulty, sexual satisfaction, distress	Not specified
VVSymQ Varicose Veins Symptoms Questionnaire	Varithena	Approved	FDA	Varicose veins	Vanish 1 and Vanish 2	11/25/2013	Symptom Improvement	Primary	For both VANISH-1 and VANISH-2, treatment with 1.0% Varithena was superior to placebo in improving symptoms as measured by VVSymQ, when either a duration or an intensity scale was used to measure patients’ symptoms
WPS-RA Rheumatoid arthritis-specific Work Productivity Survey	Cimzia	Approved	EMA	Spondylitis, ankylosing	Study AS001	10/18/2013	Productivity	Secondary	Cimzia-treated patients reported significant improvements in axial spondyloarthritis-related productivity at work and within household, as reported by the Work Productivity Survey as compared to placebo. For patients remaining in the study, improvements in all afore-mentioned outcomes were largely maintained through Week 204
				Arthritis, psoriatic	Study PsA001	11/25/2013	Productivity	Secondary	Cimzia statistically improved ACR component response rates and psoriatic arthritis-related productivity at work and within household, as reported by the Work Productivity Survey compared to placebo. Improvements in all afore-mentioned outcomes were maintained through Week 216.

Abbreviations. ACR, American College of Rheumatology; EMA, European Medicines Agency; FDA, US Food and Drug Administration; PRO, patient-reported outcomes.

Psychometric properties

Among the 20 novel PRO instruments, the psychometric properties most frequently validated were internal consistency reliability, known-group validity, responsiveness, minimal important difference (MID), and concurrent validity. For the five PROs that led to labelling claims, the psychometric properties validated were responsiveness, known group validity, MID, clinical validity, internal consistency reliability, concurrent validity, test-retest reliability, and factor analysis (supplemental Figure S1).

Discussion

Inclusion of PRO-based information on product labels is important for making patient-centered information available to decision makers in the healthcare and the pharmaceutical industry¹², as well as contributing to ongoing initiatives for increasing patient engagement and patient-focused drug-development¹⁵. Unfortunately, this information is often missing from product labels^5,11 which may be partly owing to the challenge of validating PROs prior to therapeutic confirmatory trials, as advocated by regulatory authorities⁸. However, this literature review identified cases in which psychometric (quantitative) validation of PRO instruments was conducted in the same phase 3 trial for which they were being used as a clinical endpoint, with some also successfully supporting labelling claims. Overall, 78 PRO instruments from 68 phase 3 studies were identified as including elements of psychometric quantitative validation, a quarter of which were novel. The PROLABELS database search revealed that five of the novel PRO instruments were used in label claims across therapeutic areas including oral mucositis (OMDQ)¹⁶, Peyronie’s disease (PDQ)^17,18, premature ejaculation (PEP)¹⁹, varicose veins (VVSymQ)²⁰, and rheumatoid arthritis (WPS-RA)²¹. All these PROs instruments have been used in subsequent clinical trials, and some have supported further label claims.

Overall, seven PRO-based labels/products that included ten claims using the same PROs were identified from the PROLABELS database, of which the EMA granted five product labels and the FDA granted two. Marketing authorization was subsequently withdrawn for two products in the EU at the request of the pharmaceutical company for “commercial reasons” (not caused by problems at the level of study design or results)^22,23. The FDA-granted label claims were focused solely on disease symptoms, whereas the EU-granted label claims encompassed both symptoms and higher order constructs (e.g. productivity, functioning). This mirrors findings from DeMuro et al.²⁴, who assessed PRO label claims granted by the FDA and EMA between 2006 and 2010. Of the 75 products approved by both agencies, around half (47%) had at least one EMA-granted PRO label claim compared to around a fifth (19%) that had one approved by the FDA. While most FDA-granted claims focused on symptoms, those granted by the EMA were more likely to include higher order concepts. Few (∼12%) were granted the same label claims. Although, where PRO label claims were granted by both the FDA and the EMA, there was similarity in the type of label claim. Previously published recommendations for maximizing the chances of success in obtaining a PRO-based label claims have included^6,11,12: ensuring systematic PRO data collection, a clear rationale for pre-specified endpoints, adequately powered clinical trials, and establishing stringent missing data criteria. Early engagement with regulatory agencies to discuss the role of PRO endpoints and specific measurement changes may also increase the chance of PRO-based claim approval¹².

In general, there are signs that PRO-based labelling is increasing, being included in 26.3% of new drugs approved by the FDA between 2016 and 2020, up from 20% in 2006–2015^25–27. However, a recent investigation found that use of PRO tools in studies of rare diseases has not improved over time^28,29, with 10.5% of orphan drugs having FDA-approved PRO-based labelling between 2006 and 2008, and 9.9% between 2009 and 2017²⁸. Of the 68 phase 3 studies retrieved in our literature search, five studies (7.4%) validated existing PRO tools in patients with rare diseases, including Cushing’s disease (n = 1), immune thrombocytopenia (ITP, n = 3), and idiopathic pulmonary fibrosis (IPF, n = 1). While two studies used generic PRO measures (e.g. CASA-Q for cough and sputum, EQ5D-VAS for quality of life, WHO bleeding scale), three studies used PRO instruments that had been specifically developed for the rare disease (Thrombocytopenic Purpura (ITP)-Patient Assessment Questionnaire (PAQ), The Kid’s ITP tool, and Cushing’s Quality-of-Life (CushingQOL) questionnaire). The identification of these studies may indicate how validation of PRO tools in the phase 3 setting is particularly useful for rare diseases and help overcome the difficulties and costs associated with PRO development in small patient populations³⁰.

A key strength of this research was the targeted nature of the literature and PROLABELS database searches, which allowed the identification of evidence from a large amount of literature to answer the specific question of whether PRO instruments psychometrically validated in the phase 3 trial setting could support label claims. There are also some potential limitations to the methodology, such as the literature search being limited to a single database, English language articles, and to studies published in the last 15 years. However, the limited 15-year period still allowed a broader search of therapeutic areas, while keeping the number of references for screening at a manageable level.

While this investigation has confirmed that PRO validation in a phase 3 trial setting is possible, not all phase 3 trials will be adequate for this purpose, and we cannot be certain if previous attempts to validate PRO measures have failed and were not made public. For a PRO measure to be accepted for labeling claims rigorous psychometric validation should include assessment of validity (content validity, construct validity and criterion validity), reliability (reproducibility and internal consistency), and ability to detect change in relation to Gold Standard measures of the same concept at appropriate timepoints¹. Investigators wishing to implement this strategy would benefit from further guidance from regulators or agencies to ensure that standards are met, and to increase efficiency of the resources invested in clinical research.

Conclusions

In summary, the challenge of PRO validation in the early stages of the product development cycle may act as a barrier to the use of PRO measures in pivotal trials, preventing the experience of disease and treatment impact to be viewed through the patient lens. However, the results of this targeted literature review confirm that quantitative (psychometric) validation of PRO tools is possible within the context of phase 3 trials. Furthermore, these phase 3 validated PROs also supported label claims in the USA and Europe. Additional clarification and guidance on the methodological requirements for PRO validation within the phase 3 setting will maximize the availability of patient-centered information and help provide the complete picture for healthcare decision-makers.

Transparency

 

Author contributions

All authors contributed to the study conception and design. SD and RA contributed to data collection and all authors contributed to the analysis and data interpretation. All authors commented on previous versions of the manuscript, and read and approved the final manuscript.

Acknowledgements

This study was initiated and supported by Astellas Pharma Global Development Inc. Medical writing support was provided by Tina Bristo, and Lisa O’Rourke, PhD, on behalf of Lumanity, funded by Astellas Pharma Inc. Lumanity assisted in drafting the manuscript under the direction of the authors and provided editorial support throughout its development. Medical writing support was funded by Astellas Pharma Inc. Data from this study was presented in part as a poster titled, “Patient-Reported Outcomes Validated in phase 3 Clinical Trials: A Targeted Literature Review” at the ISPOR 2022 congress on May 15–18, 2022.

Declaration of funding

This study was sponsored by Astellas Pharma Global Development Inc. Medical writing support was funded by Astellas Pharma Inc.

Declaration of financial/other relationships

AM is an employee of, and holds stocks in, Astellas Pharma Europe; SD and RA are employees of Mapi, vendors contracted by Astellas Pharma Inc; JS is an employee of, and holds stocks in, Astellas Pharma Inc.

A reviewer on this manuscript disclosed that they are a full-time employee and shareholder of WCG Inc. Peer reviewers on this manuscript have no other relevant financial relationships or otherwise to disclose.

Data availability statement

Researchers may request access to anonymized participant-level data, trial-level data and protocols from Astellas-sponsored clinical trials at www.clinicalstudydatarequest.com. For the Astellas criteria on data sharing, see: https://clinicalstudydatarequest.com/Study-Sponsors/Study-Sponsors-Astellas.aspx