Harmonizing and consolidating the measurement of patient-reported information at health care institutions: a position statement of the Mayo Clinic

Abstract

Patient-reported outcomes (PROs) capture how patients perceive their health and their health care; their use in clinical research is longstanding. Today, however, PROs increasingly are being used to inform the care of individual patients, and document the performance of health care entities. We recently wrote and internally distributed an institutional position statement titled “Harmonizing and Consolidating the Measurement of Patient-Reported Outcomes at Mayo Clinic: A Position Statement for the Center for the Science of Health Care Delivery”. The statement is meant to educate clinicians, clinical teams, and institutional administrators about the merits of using PROs in a systematic manner for clinical care and quality measurement throughout the institution. The present article summarizes the most important messages from the statement, describing PROs and their use, identifying practical considerations for implementing them in routine practice, elucidating potential barriers to their use, and formulating strategies to overcome these barriers. The lessons learned from our experience – including pitfalls, challenges, and successes – may inform other health care institutions that are interested in systematically using PROs in health care delivery science and practice.

Keywords:

• patient-reported outcomes
• questionnaire
• patient-centered
• quality of life
• health care quality

Introduction

A patient-centered institution should be aware of how their patients perceive their health and their health care. Patient-reported Outcomes (PROs) can capture these perceptions and complement clinical metrics to inform practice, assess the value of health care, and facilitate patient-centered research. An interdisciplinary team of us including researchers and clinicians recently wrote an institutional position statement on PROs and their measurement. The statement was written primarily for clinicians, clinical teams, and administrators throughout the Mayo Clinic enterprise: including tertiary-care facilities in Rochester, Minnesota, Scottsdale, Arizona, and Jacksonville, Florida, working along with a large affiliated network of community hospitals throughout the United States and abroad (the Mayo Clinic Health System). The statement is meant to educate stakeholders within these settings about the merits of collecting and reporting PROs, as well as the importance of strategically consolidating measurement throughout the enterprise. While it does draw on prior literature, the statement is not a comprehensive methodological review of PROs. For technical reviews of PROs, including their use and limitations, we refer the interested reader to several excellent recent reviews by Cella et al¹ (PRO use in performance measurement), as well as Frost et al² and Halyard et al^3,4 (PRO use in clinical practice).

The writing team consisted of individuals with expertise in the following areas: patient-reported outcome–quality of life (PRO-QOL) science, survey methods, health care policy research, health care delivery research, value/quality measurement, preventive medicine, and clinical practice (including practice re-design). In composing the statement, the team drew from its considerable experience in patient-reported outcomes, and clinical, patient-centered research.^3–29 The statement is divided into six chapters, each written by a content expert (or team of experts) in a particular area; a glossary of PRO-related terms is also provided in an appendix. In this article, we summarize some of the key messages articulated in the statement – messages that could inform other health care institutions interested in the widespread use and application of PROs. Our goal is to identify practical considerations, and potential solutions to the challenges of implementing PROs within health care settings by leveraging the experience of our institution. While each institution is certainly unique, we believe that some of the pragmatic lessons which we have learned may cut across institutions. If further detail is desired, we invite the interested reader to access the entire statement here: http://www.mayo.edu/research/centers-programs/robert-d-patricia-e-kern-center-science-health-care-delivery/patient-reported-information-at-health-care-institutions. We begin by defining a PRO and describing some of its uses.

What is a patient-reported outcome?

A PRO is any report of the status of a patient’s health condition, health behavior, or experience with health care that comes directly from the patient, without interpretation by a clinician or anyone else.¹ PRO is an umbrella term that classifies a range of different, patient-related concepts, including personal reports of health status (such as assessments of functional status), symptoms, and health-related quality of life. It also includes assessments of health-related behaviors: including both those that are detrimental to health (eg, smoking), and those that promote good health (eg, exercise). Although health behavior assessments are often considered predictors of health outcome, they can also be construed as health outcomes, inasmuch as health care interventions can have an impact on them.¹ Finally, PROs can be used also to capture the patient’s experience with health care, in the form of assessments of patient satisfaction (eg, treatment satisfaction), patient engagement (eg, shared decision making), and consumer experience with health care services (ie, quality of care).¹ Not all medical or health information collected from patients constitutes a PRO. For example, demographic characteristics, current medication lists, and personal and family medical history are all important pieces of health information that a patient may provide; however, this information does not represent a health outcome, per se, and is therefore not a PRO.

PRO information can be obtained in a variety of ways, including interviews, questionnaires (via paper, dedicated electronic device, the web, or telephone – via interactive voice response), and diaries. PROs are distinct from other measures of clinical efficacy (such as laboratory and biometric measurements) in that they are informed entirely by the subjective impressions of patients, without interpretation by any other person. Physiological measures are important indicators of patient health status. However, when used alone, they fail to provide a full profile of how a patient is functioning in their everyday life. Hence, inclusion and routine collection of PROs, in addition to traditional clinical endpoints, will provide the most complete picture of patient health status.

What are the properties of a good PRO?

A PRO is assessed using a measure or tool. Throughout this article, the term PRO refers both to the health concept being assessed, and the tool being used to measure it. Several desirable characteristics of a PRO are described below:

1. It is simple

   A PRO measure should not require reading skills beyond that of a 12-year-old (6th Grade level).³⁰ Items should be jargon-free, the rating scales intuitive, and the instructions easy to understand.

2. It is brief

   Short, patient-friendly instruments are preferred, especially for use in clinical practice.^1,31 Question sets that are too long will not be completed. It has been our experience that a question set that requires more than 12–15 minutes to complete is probably too long for routine clinical use.

3. It is informed by patients

   PROs developed using input from patients are more relevant and more meaningful to patients than those which rely on clinician and researcher input alone.³²

4. It is reliable, valid, and responsive to change

   A reliable PRO delivers responses that are reproducible and consistent (eg, stable within a brief period).^33,34 A valid PRO is one that measures what it is intended to measure and not something else.^33,34 Measures that are to be used prospectively should be able to detect underlying change in a patient’s status over time.³³

5. It is easily scored and interpreted, and predicates clinical action

   Responses to PRO measures should be easy to score, interpret, and use.² Measures that can be immediately scored are particularly desirable for practice settings, as are measures that produce scores that are actionable, and facilitative of clinical decisions.^2,35

Current and new uses of PROs

Today, PROs are being used for clinical research, clinical practice, and performance measurement. Research usage of PROs is longstanding and includes: determining the efficacy of novel interventions (ie, randomized clinical trials), comparing outcomes of established interventions (ie, comparative effectiveness research), and measuring health status in select patient cohorts (ie, observational cohort studies, population health studies). The use of PROs in clinical practice settings is a more recent development. Evidence is accumulating that routine formal assessment of PROs in the clinical setting can lead to improved patient care.³⁶ Adding PRO assessment can improve problem identification and patient–physician communication. A systematic review of randomized clinical trials on the impact of feeding back PRO information to clinicians showed that in over half of trials care processes were favorably affected.¹⁸ Specifically, PRO feedback increased the numbers of diagnoses and notations made in the medical chart, and led to more clinician advice, education, and counseling during patient visits. Some studies have also found significant increases in patient activities to manage problematic issues and in the number of times quality-of-life issues are discussed during consultations.^37–40

There is now increasing interest in using PROs for the purposes of accountability and performance improvement. A PRO can be used to inform derivation of a performance metric, known as a PRO-based performance measure or PRO-PM.¹ A PRO-PM can be based on the scores of a PRO measure alone, or in combination with other clinical data (eg, a diagnostic code). Unlike using PRO data to inform the care of individual patients, PRO-PM data are aggregated for an entire health care entity. For example, a PRO-PM could consist of the percentage of patients in an accountable care organization who have an improved depression score, measured using a standardized PRO tool.¹ Using PROs in this way could help address a key challenge in health care reform: achieving the highest quality care, at the lowest possible cost. But in order for this to be realized, two major challenges must first be met.¹ First, PROs have not been widely adopted for use outside of research; hence, they are unfamiliar to many clinicians, payers, and providers. Second there is little known about the best set of questions to aggregate for the purpose of measuring performance of a health care entity. The lack of recognized standard PRO metrics precludes performance comparisons. Efforts are currently underway to determine guidelines and best practices for selecting PROs and aggregating PRO data into PRO-PMs.^1,41

Considerations for implementing PROs into clinical practice

Any system designed for collection of PRO data should be based on the practical considerations below.³

1. What is the purpose of data collection?

   Consideration should be given to whether the data are to be used strictly for clinical practice or whether there will also be a desire to mine the data at a later time, for research purposes or performance measurement. Both near- and long-term data needs should be considered.

2. What are the system design considerations?

   The design steps for implementing a PRO system should include agreed upon goals for PRO collection, development of system specifications, feedback about design (from clinicians and patients), website construction, and usability testing with patients. For some institutions, the use of an outside vendor is a viable option, especially where internal IT resources have a backlog of demands. The ideal vendor would work closely with clinicians and staff at the host institution to develop practical, efficient means of collecting PRO data and feeding it back to end users. Feasibility testing of the system in a busy clinical practice should be carried out prior to widespread implementation as should a formal assessment of a new system’s compatibility with existing IT infrastructures.

3. How will data collection occur?

   When PRO-QOL data are to be collected in the clinical practice setting, several factors should be considered.⁴²

   • The method of data collection, given available resources, is important and will often determine such basics as whether paper or some form of electronic media (eg, desktop computer, tablet or mobile device, interactive voice response system) will be used. Ideally, the PRO information should be collected electronically, as this obviates the need to scan paper forms into the medical record and allows for the efficient viewing of scores over time, as currently is done with laboratory data.

   • Web-based PRO systems ideally should integrate with an institution’s existing electronic health record (EHR), to facilitate use during the clinical visit. In feasibility tests at Mayo Clinic (Arizona), we found that housing PRO data separately from the medical record is unpopular with clinicians as it forces multiple logins to different systems.

   • Electronic data collection allows patients to enter data remotely (eg, from home), and can enable electronic reminder alerts (eg, text message, email).

   • Regardless of the mode of delivery, patients must be instructed about how to complete questionnaires. Most patients, including the elderly, are receptive to electronic data capture.

   • It is important to determine the timing of data collection. For many practices, it is most convenient to have the patient complete a predetermined questionnaire at the time of their visit. Other practices prefer to have data collected prior to the visit. Rules for the timing of any follow-up assessments should be specified and will likely vary with the practice, and the purpose of the assessment.

4. What PRO instruments will be used?

   Issues to consider when selecting which PRO instruments to use include the following:⁴³

   • Whether to use generic or disease-specific questionnaires, profile- or preference-based measures, single- or multi-item scales, and static (where every patient responds to the same set of questions) or dynamic questionnaires (where questions sets are tailored to the individual, as in computer adaptive tests).

   • Balancing clinician and patient preferences regarding the types of data to collect.

     • ○ Physicians may want only to assess issues they know how to treat or manage.

     • ○ Patients may have specific symptoms and quality of life (QOL) issues that they wish to discuss with the clinician.

   • Balancing data collection on a range of important issues, against patient burden in filling out questionnaires.

5. Who will review the data and when?

   Regardless of when the data are collected they should be reviewed either before or during the patient encounter. Who will conduct the review and subsequently address the PRO issues with the patient must be determined a priori. In many instances, it is the physician overseeing care who first engages the patient. However, other members of the clinical team (eg, nurses, social workers, psychologists, physical therapists) may also find this information helpful when interacting with the patient and formulating care plans. Clinicians should be willing to discuss some, if not all, of the clinically significant issues that were identified by the patient, if the patient expresses a desire to do so. Clinical staff should receive some training in the interpretation of the PRO data (see the following section).

6. How can scores be interpreted easily and made clinically relevant? What will be done with the collected data?

   The following are ways to make PRO scores interpretable for clinical use:

   • Scoring guidelines provide information on the meaning of scores (eg, “higher scores mean better functioning”).

   • Cut-off scores for clinical “caseness” (or levels of severity, eg, no disability, moderate disability, severe disability). Cut-off scores are available for some measures that target specific clinical conditions (eg, depression, chronic pain), but are not universally available for all PROs.

   • Reference scores from research studies with similar patients – from the general population with the same condition (condition-specific norms), or from healthy populations (general norms) – can be a useful way to benchmark individual scores.^44,45

   • Instrument-specific, minimally important difference estimates (MIDs), if available, or any other instrument-specific guidance on what scores constitute clinically significant changes.²

It is important that clinicians are able to interpret PRO data easily, and that the data are clinically relevant. Brief training of clinicians can be undertaken to facilitate score interpretation. Also, obtaining clinician input on the design of the data display is important.⁴⁶ For patients, the best method for presenting data has yet to be determined. One study has shown that the presentation of QOL information in simple graphs or written texts is preferred over more complex graphical information.⁴⁷ Line graphs are rated highest in both their ease of interpretation and perceived level of helpfulness.

PRO data are most useful when scores are translated into clinical actions. In a general sense, there are at least five potential uses for PRO data in the clinical practice. These include; 1) screening for problems; 2) monitoring health status; 3) identifying patient preferences, to assist clinicians in making informed decisions; 4) improving patient-provider communication; and 5) facilitating shared decision making between patients and providers.^48,49 A relevant example from our own experience serves to illustrate. At our radiation oncology clinic at Mayo Clinic (Arizona), a pilot, using real-time capture of PROs, was carried out to test the feasibility of integrating PRO collection into practice. Health care personnel reviewed patients’ PRO scores with them, to clarify and elaborate on any problems identified by the assessment. A pilot is currently being implemented at Mayo Clinic (Rochester), utilizing lessons learned in Arizona, to evaluate web-based collection of PRO data outside of the clinic, for use at the time of the oncology clinic visit.

Barriers to implementing PROs across a large, multi-specialty institution

Our experience has also highlighted several barriers to implementing a harmonized and cohesive PRO system across a large, multi-specialty health care institution. Below are a few critical barriers that we are currently dealing with.

• Efforts to collect PROs tend to be fragmented with existing systems developed within the confines of specific practices. Historically, there has been no single entity to coordinate and oversee these efforts. The result is a wide variety of PROs being used across the institution, including a fair share of “home grown” measures, which limits the ability to aggregate or compare results.

• Currently, there is no unifying institutional policy or strategy that stipulates the collection, reporting, and use of PROs.

• A strong patient-focused rationale for collecting PROs is lacking. Most efforts appear to be motivated by the need to comply with accreditation requirements, outside initiatives (eg, state-wide programs and mandates), or reimbursement from payers. This has little to do with achieving more patient-centered care.

• There is some clinical inertia to collecting PROs. Some clinicians are concerned that routine collection of PROs will be too administratively burdensome, that it will disrupt the clinic workflow, that patients will not want to do them, and that they will take up too much time during patient encounters.

• Clinicians may simply choose not to use (or even look at) the information gathered by PROs. Regardless of how relevant PROs are to a patient’s situation, their widespread adoption will not occur if the information is never used by clinicians.

• The search for the perfect PRO measurement system has been a major challenge. In this sense, the search for the perfect is truly “the enemy of the good”. While it is challenging to achieve in a large health care institution, a spirit of cooperation across specialties, practices, and physicians must be embraced. There is no perfect system that will meet all the needs of every clinician, for every patient, at every encounter. If development of a perfect system is the expectation, the patient’s voice will never be heard to the extent that it can be, and opportunities for achieving more patient-centered care will be lost.

• There is incomplete integration of PROs into the EHR. In many instances, PRO data are simply attached as a clinical note, with limited guidance as to how to interpret them.

Strategies for overcoming the barriers

Despite the challenges imposed by these barriers, they can be overcome. The following are a few strategies that we feel may support the implementation of PRO data capture across a large health care institution:

Strategy 1: define entities to help coordinate PRO efforts across the institution

At Mayo Clinic, this includes both practice and scientific entities working in concert. The practice-based Patient Clinical Questionnaires Approval Committee is charged with reviewing and prioritizing all candidate questionnaires and surveys put forth for institution-wide usage. Mayo Clinic’s recently-formed Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery is available to provide scientific and IT expertise to facilitate selection and inclusion of scientifically robust PROs into the practice.

Strategy 2: identify clinical champions who can both facilitate the implementation of PROs and assure that the data collected are meaningfully used

Such clinicians should be committed to working collaboratively with PRO methodologists, IT personnel, and other administrative stakeholders.

Strategy 3: leverage IT support to enable complete integration of PRO data directly into the EHR

Our prior experience has shown that indirect linking of PROs to the EHR is a major disincentive to clinician use. Ideally, the PRO data would be stored in the EHR in an elemental form to allow for easier tracking, transfer, and computation. Scanning data collection forms into the EHR is much less preferred.

Strategy 4: articulate an institution-wide commitment to use at least some standard, psychometrically-sound PRO tools when appropriate and feasible

Preferably, these tools will be non-proprietary, in the public domain, and provide scores that can be benchmarked against other institutions. These tools must be readily accessible and immediately available throughout the institution.

Strategy 5: provide interpretation guidelines

PRO methodologists can provide guidance on how to interpret scores and patient responses from selected measures, provided that it is available from published guidelines, or directly from the developer of the measure. If unavailable, PRO methodologists can provide clinicians with suggestions to aid score interpretation.

Success stories and positive steps

Mayo Clinic has achieved some limited success in implementing PROs and other patient-reported information systems. Since November 2009, single-item measures of pain, fatigue, and overall QOL have been collected from patients seen in the Division of Hematology (a mix of cancer and non-malignant hematology patients), and noted in the EHR. Patients complete these measures at the time that they are roomed for their appointment. Clinicians are provided with suggested cut-off scores for poor function, and resources for clinical action, to help make optimal use of the patient data. In 2010, the Department of Medical Oncology adopted the same three measures for use with cancer patients seen in-clinic.

Since 1995, a limited amount of patient-reported health data has been collected at every patient visit, using Mayo Clinic’s standard Patient Provided Information (PPI) forms. At a minimum, information on a patient’s current symptoms, medications, allergies, substance use history, self-care ability, home environment, and personal and family medical history is collected. Decision rules are used to determine if any additional data forms are needed for specific patients (eg, depression screener, asthma control test, etc). They can be completed in paper form or online, through Mayo’s patient portal. Data are scanned into the patient’s EHR. While utility of the information at the point-of-care has been questioned by some patients and providers, the PPI system does demonstrate the feasibility of collecting patient-reported information across the institution.¹⁵

Recently, several of us developed a brief PRO-QOL tool to systematically capture key patient important concerns for use by clinicians and provider teams. The tool was developed within a Southeast Minnesota Beacon Community demonstration project sponsored by the Office of the National Coordinator for Health Information Technology, Department of Health and Human Services. Beacon grants were awarded to 17 communities throughout the US, to facilitate greater health IT integration and data sharing within a targeted community. The tool feeds the collected patient data to the clinician in real-time, for use during the visit and to aid care planning. It enables patients to report on concerns that may be less frequently discussed during routine visits, such as barriers to self-care, personal relationships, emotional health, and financial concerns. Development of PRO-QOL was driven by input from patients and providers in order to maximize acceptance. The current version of the tool is specified for diabetes; however, tailoring for other disease contexts and for patients with multiple chronic health conditions is being planned. Complete details of the development of Beacon PRO-QOL can be found in Ridgeway et al.²⁴ The initial screenshot and a sample domain checklist are displayed in Figure 1.

Figure 1 The Patient-Reported Outcome-Quality of Life Tool.

Note: Reproduced from Ridgeway JL, Beebe TJ, Chute CG, et al. A brief patient-reported outcomes quality of life (PROQOL) instrument to improve patient care. PLoS Med. 2013;10(11):e1001548. © 2013 Ridgeway et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Finally, we have made progress in identifying at least one standard PRO measure for use across the institution; one that would allow benchmark comparisons with other health care institutions. After a recent vetting of general health status measures, the Mayo Clinic Kern Center for the Science of Health Care Delivery made the decision to endorse the PROMIS Global-10 measure as an institutional standard for assessing general physical and general mental functioning. Results of this vetting process including the criteria used for comparing candidate measures are documented in chapter 4 of the Position Statement (see http://www.mayo.edu/research/centers-programs/robert-d-patricia-e-kern-center-science-health-care-delivery/patient-reported-information-at-health-care-institutions). Other centers and integrated health systems are currently using the Global-10. It is the general measure of choice in most of the participating medical centers involved in Dartmouth’s patient-reported measures initiative (see http://tdiprm.dartmouth.edu/overview.php), including the High Value Healthcare Collaborative (see http://highvaluehealthcare.org/). It is also being used in diabetes and heart specialties at Brigham and Women’s and Massachusetts General Hospitals (the Partners Healthcare group).⁵⁰ General physical and mental health component scores of the Global-10 have been assigned codes using Logical Observation Identifiers Names and Codes (LOINC), which enables electronic health information exchange; and the measure will qualify as a clinical quality measure for Meaningful-Use stage 2 within the Center for Medicare and Medicaid Services EHR incentive program beginning in 2014.⁵¹

Conclusion

Patient-reported outcomes (PROs) have a long history of use in research. However, today they are being used in other contexts as well. At an individual patient level, PROs are being used clinically to provide physicians and other providers with valuable information about a patient’s health status – information that can help screen for problems, monitor health over time, and improve patient-provider communication. At an aggregate level, data from PROs are being combined with other clinical data to form performance metrics (PRO-PMs) that enable outcome comparison of health care entities (eg, practices, hospitals, health care plans). While clinical use of PROs may require some tailoring to the needs of an individual practice, a PRO-PM requires acceptance of at least some standard PRO measures that can serve as a basis for comparison across health care entities (or allow for score “cross-walking” to such standard measures). Hence, a hybrid approach will likely be needed that combines customized, practice-informed PRO assessments with psychometrically-robust, readily-available, and easily-benchmarked standard PROs. The former will involve intramural collaborations between PRO methodologists and clinical practice champions. The latter will involve cross-institution collaboration, and agreement on suitable standards. Our experiences outlined in this article can provide some valuable lessons that may inform a roadmap for attaining both of these ends. However, we do recognize that individual health care institutions may have their own unique set of challenges in implementing PROs. Hence, while our experience does provide guidance, it does not necessarily provide a single “best” solution toward harmonizing and consolidating PROs for every health care institution. We hope that the present article provides at least some insight as to how system-wide use of PROs can be achieved. In today’s increasingly competitive health care environment, the consistent use and effective integration of PROs may be one way that an institution can “win over” consumers with a commitment to more compassionate, patient-centered care.

Acknowledgments

This work was supported by the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery. The views expressed by the authors have been endorsed by the center, but do not necessarily reflect the views of all Mayo Clinic professionals and staff. The authors would like to thank Dr David Cella for important feedback on the use of general health status measures for performance measurement in health care settings.

Disclosure

The authors report no conflicts of interest in this work.