Patient-reported outcome measures in arthroplasty registries

Abstract

Abstract — The International Society of Arthroplasty Registries (ISAR) Patient-Reported Outcome Measures (PROMs) Working Group have evaluated and recommended best practices in the selection, administration, and interpretation of PROMs for hip and knee arthroplasty registries. The 2 generic PROMs in common use are the Short Form health surveys (SF-36 or SF-12) and EuroQol 5-dimension (EQ-5D). The Working Group recommends that registries should choose specific PROMs that have been appropriately developed with good measurement properties for arthroplasty patients. The Working Group recommend the use of a 1-item pain question (“During the past 4 weeks, how would you describe the pain you usually have in your [right/left] [hip/knee]?”; response: none, very mild, mild, moderate, or severe) and a single-item satisfaction outcome (“How satisfied are you with your [right/left] [hip/knee] replacement?”; response: very unsatisfied, dissatisfied, neutral, satisfied, or very satisfied). Survey logistics include patient instructions, paper- and electronic-based data collection, reminders for follow-up, centralized as opposed to hospital-based follow-up, sample size, patient- or joint-specific evaluation, collection intervals, frequency of response, missing values, and factors in establishing a PROMs registry program. The Working Group recommends including age, sex, diagnosis at joint, general health status preoperatively, and joint pain and function score in case-mix adjustment models. Interpretation and statistical analysis should consider the absolute level of pain, function, and general health status as well as improvement, missing data, approaches to analysis and case-mix adjustment, minimal clinically important difference, and minimal detectable change. The Working Group recommends data collection immediately before and 1 year after surgery, a threshold of 60% for acceptable frequency of response, documentation of non-responders, and documentation of incomplete or missing data.

The International Society of Arthroplasty Registries (ISAR) Patient-Reported Outcome Measures (PROMs) Working Group was established to convene, evaluate, and advise on best practices in the selection, administration, and interpretation of PROMs for hip and knee arthroplasty in registries worldwide. This report reviews the Working Group recommendations for selection, administration, and analysis of PROMs by arthroplasty registries.

Lessons learned from patient-reported outcome measures in arthroplasty registries

Patient-centered outcomes reflect the current change of focus from volume-based to value-based healthcare delivery. Measuring patient-centered outcomes has provided important information about outcomes that matter to patients. These outcomes confirm that joint replacement is an effective treatment for disabling hip or knee joint diseases. In addition, these outcomes provide information about an important minority of patients who do not improve as expected or express dissatisfaction with the results of the intervention (Baker et al. 2007, Rolfson et al. 2011a).

Data from arthroplasty registries have identified several patient-related determinants of pain relief, functional outcomes, and satisfaction such as emotional health (Franklin et al. 2008, Rolfson et al. 2009), presence of coexisting low back pain, and other medical conditions that impair ambulation (Ayers et al. 2013, Gordon et al. 2014b), socioeconomic status (Greene et al. 2014a), and obesity (Jameson et al. 2014b). This information is not intended to give reasons to exclude patients from having joint replacement. However, these variables should be considered in the shared decision-making process to inform patients and give them realistic expectations.

The existing PROMs programs have provided knowledge about the association between surgical factors such as surgical approach and patient-reported outcomes (Jameson et al. 2014a, Lindgren et al. 2014). Furthermore, registry data have shown that low function after hip replacement is a risk factor for having a subsequent revision (Devane et al. 2013). However, considering the large amount of literature about PROMs in joint replacement from registries and clinical studies, it is important to recognize that little published information is available on how PROMs may be used in clinical practice to improve results.

Patient-reported outcome measures in hip and knee replacement

Generic patient-reported outcome measures

There are 2 primary surveys used in joint replacement registries to measure health-related quality of life (HRQoL): the Short Form health surveys (Optum Inc. 2015) and the EuroQol health outcome measures (EuroQol Group 2015) (Table 1). The Short Form 36 health survey (SF-36) includes 8 dimensions of health that are summarized into 1 physical and 1 mental scale component (Ware et al. 1992). The SF-36 is the most commonly used generic PROM in clinical trials, and is psychometrically sound for patients who have osteoarthritis (Kosinski et al. 1999). However, for routine follow-up in joint replacement registries, the SF-12, which is a shortened version of the SF-36, has often been preferred (Ware et al. 1996, Rolfson et al. 2011b). Although Short Form tools require licensing, the equivalent Veterans Rand 12-item survey (VR-12) and 36-item survey (VR-36) are available for non-commercial use without charge (Boston University School of Public Health 2015).

Table 1. Generic and specific patient-reported outcome measures commonly used in hip and knee arthroplasty^a

Type Name of survey	Abbreviation for survey	Year developed	Registries using the survey	Validated for arthroplasty	License requirements	No. of translations available^b	No. of items	Minutes to complete survey
Generic
EuroQol 5 dimension health outcome survey,
3-level version	EQ-5D-3L	1990	7	Hip and knee	Yes	> 170	6	1–2
5-level version	EQ-5D-5L	2011	0	Unknown	Yes		6	2–3
Short Form 36 health survey	SF-36	1992	1	Hip and knee	Yes	> 50	36	5–10
Short Form 12 health survey	SF-12	1996	4	Unknown	Yes	> 40	12	2
Short Form 6-domain health survey	SF-6D	1998	0	Unknown	Yes	?	6	1–2
Specific
Western Ontario and McMaster Universities Arthritis Index	WOMAC	1982	4	Hip and knee	Yes	92	24	5–10
Knee injury and Osteoarthritis Outcome Score	KOOS	1998	7	Knee	No	44	42	5–10
Hip disability and Osteoarthritis Outcome Score	HOOS	2003	4	Hip	No	17		5–10
KOOS Physical Function Short Form	KOOS-PS	2007	0	Unknown	No	15	7	1–2
HOOS Physical Function Short Form	HOOS-PS	2008	0	Unknown	No	9	5	1–2
Oxford Knee Score	OKS	1998	3	Knee	Yes	19	12	3–4
Oxford Hip Score	OHS	1996	3	Hip	Yes	11	12	3–4
University of California at Los Angeles Activity Score	UCLA	1984	4	Hip and knee No		?	1 (10 levels)	2–3

a Domains covered in surveys:

EQ-5D-3L, EQ-5D-5L: mobility, self-care, usual activities, pain/discomfort, anxiety/depression.

SF-36, SF-12, (and the similar Veterans Rand 12 (VR-12) and Veterans Rand 36 (VR-36)): vitality, physical functioning, bodily pain, general health perceptions, physical role functioning, emotional role functioning, social role functioning, mental health.

SF-6D: physical functioning, role-participation (combined role-physical and role-emotional), social functioning, bodily pain, mental health, vitality.

WOMAC: pain, disability, and joint stiffness in knee and hip osteoarthritis.

KOOS/HOOS: pain, other symptoms, function in activities of daily living (ADL), function in sport and recreation (Sport/Rec), knee-/hip-related quality of life (QOL).

KOOS-PS/HOOS-PS: function, daily living, sport, recreation.

OKS, OHS: joint pain and function.

UCLA: level of activity.

b Translation of a validated PROM into different languages requires a formal methodological approach and specific expertise. Not all translated versions of these surveys have been validated.

The EuroQol 5-dimension (EQ-5D) is a generic measure of health status developed by the EuroQol Group (EuroQol Group 1990). The survey includes 5 health outcome domains or items that can be summarized into a utility score, and an accompanying visual analog scale (VAS) that addresses current health state (EQ VAS). There are several different value sets to calculate the utility scores, and each value set represents the preferences of the population from which it was derived. Thus, comparisons of results using utility indices calculated with different value sets may be difficult (Gordon et al. 2013). Differences in methods for establishment of value sets may explain differences between values sets more than differences in preferences of the populations in which they were developed.

The EQ-5D-5L is an extended version of the EQ-5D that has 5 response options for each dimension (Herdman et al. 2011). The original EQ-5D, which has 3 levels of response options (EQ-5D-3L), is most commonly used and has been validated for patients with osteoarthritis (Barton et al. 2009, Obradovic et al. 2013). However, the EQ-5D-5L has better psychometric properties (such as better responsiveness and lower ceiling effects) than the EQ-5D-3L, and increased use of the EQ-5D-5L is anticipated in clinical studies and registries (Greene et al. 2014b).

Both Short Form and EuroQol tools are commonly used, and there are no strong advantages of one tool over the other. Thus, the ISAR PROMs Working Group does not make specific recommendations about the preferred generic PROMs tool. A crosswalk algorithm is available to convert SF-12 responses to EQ-5D index scores, which may enable comparisons between the tools (Le 2014).

Specific patient-reported outcome measures

Numerous comprehensive specific PROMs instruments are available for patients who have hip or knee problems (Table 1). These measures typically encompass several outcome domains. The Western Ontario and McMaster Universities Arthritis Index (WOMAC) assesses pain, disability, and joint stiffness in patients who have hip or knee osteoarthritis (Bellamy et al. 1988). This 24-item questionnaire, with >90 translations available, has been proven to be valid, reliable, and responsive to osteoarthritis outcomes and is commonly used in clinical trials (Bellamy et al. 1988, McConnell et al. 2001). A 12-item version including all 5 pain questions of the original full questionnaire and 7 of the disability questions has also been developed and validated in a relevant osteoarthritis population (Whitehouse et al. 2003). The WOMAC requires licensing, and is used in some large local and regional registries in the USA.

The non-proprietary Knee injury and Osteoarthritis Outcome Score (KOOS) and Hip disability and Osteoarthritis Outcome Score (HOOS) were developed as extensions and comprehensive alternatives to the WOMAC (Roos et al. 1998, Nilsdotter et al. 2003). The KOOS and HOOS questionnaires are being used in some United States and European registries. However, they may be burdensome for respondents because of the large number of questions (KOOS, 42 questions; HOOS, 40 questions). Thus, the KOOS and HOOS may not be appropriate for use in routine follow-up programs.

The KOOS Physical Function Short Form (KOOS-PS, 7 questions) and HOOS Physical Function Short Form (HOOS-PS, 5 questions) are short versions of the KOOS and HOOS that include physical function items but not pain items (Davis et al. 2008, Perruccio et al. 2008). Despite their brevity, the KOOS-PS and HOOS-PS have good measurement properties for physical functioning in patients with hip and knee osteoarthritis. These surveys do not require licensing and have been translated into many languages (total: KOOS-PS, 15 languages; HOOS-PS, 10 languages). Although the KOOS-PS and the HOOS-PS are not established measures in joint replacement registries, these short questionnaires have been included in the International Consortium for Health Outcomes Measurement (ICHOM) hip and knee osteoarthritis standard set of outcomes measures (International Consortium for Health Outcomes Measurement 2015).

The commonly used joint-specific, 12-item, Oxford Knee Score (OKS) and Oxford Hip Score (OHS) address pain and function, and were designed from interviews with patients to measure outcomes after knee or hip joint replacement surgery (Dawson et al. 1996a, 1998, Murray et al. 2007). These surveys are available in several languages (OKS, 20 languages; OHS, 10 languages) and are useful in clinical studies and joint replacement registries (Browne et al. 2013, Devane et al. 2013). The OKS and OHS require licensing, are available without charge for non-commercial use, and are used in registry PROMs programs in the UK and New Zealand.

Although some PROMs instruments for hip or knee problems include pain measures, joint pain is often measured with a VAS or numeric rating scale (NRS) in isolation. Despite differences in granularity, possible modes of collection, and presentation requirements, VAS and NRS have an acceptable correspondence (Breivik et al. 2000, Hawker et al. 2011, Hjermstad et al. 2011). Manual readings of VAS scores on pen-and-paper questionnaires require more work than the NRS, and are more susceptible to transcription errors. With VAS surveys that require the respondent to mark a point on a line, it may be difficult to standardize the length of the line for different modes of data collection, especially when some patients complete the survey eletronically and other patients use a hard-copy questionnaire. Furthermore, some patients may have conceptual difficulty in completing the VAS (Wewers et al. 1990).

There is variation in the specific PROMs instruments that are used in joint replacement registries. Thus, the ISAR PROMs Working Group does not make specific recommendations about which PROMs to use in arthroplasty registries. However, the Working Group recommends that registries should choose PROMs instruments that (1) were appropriately developed with a relevant patient population, and (2) have evidence of good measurement properties for patients who have arthroplasty.

Pain is considered the major construct of interest. So, for the purpose of harmonization, the Working Group agreed about recommending a 1-item pain question. The suggested measure was slightly modified from the first item of the OHS and OKS (“During the past 4 weeks, how would you describe the pain you usually have in your [right/left] [hip/knee]?”), which has 5 Likert-graded response options (none, very mild, mild, moderate, and severe). The original question is available in many languages and may be easily translated into other languages. The question has good measurement properties, validity, and reliability (Dawson et al. 1996b). It is simple to enter data from pen-and-paper questionnaires with responses to this question, instead of the difficulty in reading a VAS, and the question enables presentation on different platforms.

Single-item satisfaction outcome measure

A single question to assess the outcome of hip or knee replacement is attractive because it is simple, easy to use, and may capture the essentials of outcomes after hip and knee arthroplasty. Although the goals of surgery include pain relief and functional improvement, the importance of these goals may vary because each patient may have specific personal expectations from surgery (Mancuso et al. 1997, 2001). A single item such as “satisfaction with the result of surgery” is conceptually appealing, has good face validity, and enables patients to express their unique points of view about the outcome of surgery.

Although it is well established that there is a correlation between satisfaction and objective improvements in pain, function, and general HRQoL (Robertsson et al. 2000, Mahomed et al. 2011, Judge et al. 2012a, Baker et al. 2013, Clement et al. 2013), satisfaction with the outcome of joint replacement may be affected by preoperative patient characteristics such as expectations (Mancuso et al. 1997, Noble et al. 2006, Hamilton et al. 2013), function (Hamilton et al. 2013), depression (Brander et al. 2007), and catastrophizing (Riddle et al. 2010). The occurrence of a postoperative complication may negatively affect overall satisfaction with surgery (Kay et al. 1983, Bourne et al. 2010). Postoperative satisfaction may therefore incorporate multiple and varied factors that affect the outcome of surgery.

There are drawbacks to using a single item to assess outcome. A single qualitative item may not measure the magnitude of change, which can be accomplished only by comparing pre- and postoperative measures of health status or by using a potentially less reliable health transition question. The response to a single qualitative item may also be affected by age, sex, and diagnosis, and it may be necessary to control for these factors for comparisons across regions (Kay et al. 1983, Robertsson et al. 2000, Merle-Vincent et al. 2011). Furthermore, it is not possible to perform cost-effectiveness analysis using a satisfaction question. Nevertheless, the benefit of using a single question may include a significantly higher frequency of response than with more comprehensive questionnaires (Robertsson et al. 2001). Published satisfaction frequencies are available from large samples (Robertsson et al. 2000, Garellick et al. 2009, 2010), but the results may be affected by the different ways in which the question is asked and the timing relative to the date of surgery.

When implementing a single-item satisfaction measure, the ISAR PROMs Working Group recommends the wording: “How satisfied are you with the results of your [right/left] [hip/knee] replacement?” When this question is asked less than 1 year after surgery, the qualification “so far” should be added, because clinically relevant improvement may occur between 6 and 12 months after surgery (Browne et al. 2013). The preferred response scale may include a range of options (very dissatisfied, dissatisfied, neutral, satisfied, and very satisfied) instead of a VAS to provide a clearer interpretation of the responses. The option “neutral” may be selected by patients who are uncertain or do not want to appear negative in their assessment; so, when reporting the overall proportion of patients who are satisfied, we recommend categorization of results from patients who respond “satisfied” or “very satisfied” as satisfied and other response options as dissatisfied.

Survey logistics

Instructions for patients

For data collection and optimal frequency of response, it is necessary to inform the patient about the needs and goals of PROMs. The patient should be given instructions or a patient guide. Motivation of all orthopedic specialists is needed for the success of PROMs collection. Ideally, orthopedic specialists should inform their patients about the PROMs collection before and after joint replacement. However, the logistics in PROMs collection should not depend on orthopedic specialist engagement in the collection of PROMs. Giving information and instructions, and collecting surveys in the clinic should preferably be done by staff members specifically assigned to the task.

Paper-based and electronic-based data collection

The PROMs may be collected and scored on paper and/or electronically. Paper-based methods of collecting PROMs data may be simple, but it is important to maintain consistency in presenting the PROMs questionnaires in the same order for every collection. After the forms are completed, they should be reviewed for completeness or missing data. When the paper questionnaire is completed in the clinic, the staff members should review the questionnaire for missing data before the patient’s departure to maximize completed items. When the paper questionnaire is mailed to the hospital, it is difficult to ensure completeness of the questionnaire, and missing items must be accepted unless there is a mechanism to contact the patient for the missing items.

After the paper survey has been completed, the data should be entered in an electronic form. Manual data entry may be susceptible to human error, but electronic rules can be constructed to allow only plausible responses. Double data entry may reduce error but it is time-consuming. Alternatively, data may be entered using a specialized scanner and software, which may have disadvantages of cost, the need for training of staff members, and the limitation that some questionnaires still may require manual data entry.

With paper PROMs forms, personnel must apply identifiers and sort, distribute, and collect the PROMs. It may be simple to collect paper forms during routinely scheduled visits to the clinic, but data collection with paper forms between clinical visits may be difficult logistically. Furthermore, bias may be introduced because patients who have problems may be more likely to return for clinic visits and may return to the clinic more frequently (Dawson et al. 2010). Thus, it is recommended that follow-up questionnaires should be sent by post as close as possible to the due data.

Electronic versions of PROMs may be difficult to implement because additional instructions may be needed for the electronic format. The wording, punctuation, and response options of the PROMs instrument should be reproduced accurately in the electronic version. The electronic survey must allow patients to go back and change previously answered questions, as with the paper version. Electronic PROMs may be collected on various hardware platforms including desktop computers, laptop computers, tablets, or handheld devices using web-enabled survey tools. For electronic collection using tablets or handheld devices in the clinic, patients can be given the device with the PROMs instrument selected upon arrival, and they can select their answer to each item as presented. Alternatively, access to web-enabled survey tools can be provided at confidential computer stations in clinic waiting rooms or through links that are e-mailed to a patient’s own device.

Electronic data collection may be more efficient and less time-consuming than paper collection because electronic versions avoid the need for manual data entry and they enable immediate scoring. However, electronic PROMs may lack equivalence with paper PROMs because electronic PROMs require access to a digital system and may not be acceptable to all patients. Furthermore, in contrast to paper PROMs, electronic PROMs are often configured to require responses to all items to complete the questionnaire, so patients must provide a response to each item before they can continue to the next item.

Some registries provide the option of either paper- or electronic-based PROMs. In the past, PROMs were measured only with paper questionnaires. More recently, institutions and registries have used electronic surveys. Patients are becoming more familiar with digital platforms than previously, but digital methods are not currently acceptable to or feasible with all patients. Advantages of using both methods may include an increase in patient compliance and a decrease in missing or incomplete data. Although disadvantages may include the increased administrative challenges of maintaining both paper- and electronic-based methods, the ISAR PROMs Working Group recommends providing patients the option of paper or electronic PROMs (Rolfson et al. 2011c, Gliklich et al. 2014). However, the Working Group acknowledges that this approach may not be feasible for logistic and economic reasons in emerging registries.

Given the rapidly transitioning technology environment (smartphones are becoming ubiquitous in younger populations and more than 70% of baby-boomers (those born in the latter half of the 1940s) use a laptop or desktop computer on a daily basis (MArketingCharts, 2015)). As baby-boomers are now reaching their late 60s, the need for paper administration will diminish with time.

Reminders for follow-up

Automatic reminders may improve efficiency and timing of follow-up. Reminders may be created by connecting PROMs instruments with clinical data such as date of surgery. For paper-based PROMs, the reminder could be sent by hospital personnel. With some systems, electronic-based PROMs questionnaires may be sent automatically to patients. It is recommended that a reminder to non-responders be sent 2 weeks after sending the initial questionnaire. In addition, it is recommended that automatic blocking steps should be programmed in to avoid inappropriately sending follow-up PROMs instruments to patients who have withdrawn their informed consent or have died.

Centralized systems or hospital-based follow-up?

Follow-up PROMs may be administered by a centralized system such as an arthroplasty registry or national health service department. With centralized systems, paper or electronic questionnaires are sent to patients and data entry is managed at a central office. Alternatively, hospitals may assume responsibility for all aspects of data collection. With hospital-based systems, members of the hospital staff send the questionnaires to patients, collect data at clinic visits, and perform data entry. The advantages of a centralized system include a uniform method of communication with patients about PROMs. However, hospital-based systems have the advantage of the relationship with the patient, and some patients may be more willing to respond to a questionnaire from the local hospital or the surgeon’s office than one from a centralized agency. Legislation and funding may determine the organization of PROMs collection in different countries.

All patients or a select sample?

Depending on the goals of the PROMs assessment and available resources, the PROMs may be used to measure the entire patient population or a select sample. When PROMs are used for quality improvement, all patients having arthroplasty should be assessed, and a sample of the population from a hospital may be inadequate for analysis. In contrast, a sample of patients may be adequate for research purposes when it is a random, unbiased sample that has a sufficient number of patients for analysis and drawing of conclusions.

Evaluation of patients or joints?

Joint registries would usually organize data collection by primary intervention, joint, and laterality, and each joint that had a primary intervention would yield a new case. This joint-based approach may be suitable for implant surveillance and procedure-related outcomes studies, and most registry PROMs programs have used this approach. However, the approach may have disadvantages for evaluation of PROMs in patients who have multiple arthroplasties, because evaluation of each joint separately may cause too heavy a load in terms of questionnaires. In addition, the use of PROMs is expanding in disciplines other than hip and knee replacement, such as oncology and psychiatry, and this may also cause an increased questionnaire load for some patients.

The alternative is to measure outcomes with a condition- and patient-centered approach, without focusing on a specific joint. The Function and Outcomes Research for Comparative Effectiveness in Total Joint Replacement (FORCE-TJR) Registry measures pain for both hips and knees at all follow-up visits and focuses the specific PROMs on the most problematic joint. This issue must be considered in PROMs collection by registries. A condition- and patient-centered approach, as opposed to joint-centered, may enable longitudinal measurements in integrated care pathways that include early stages of hip and knee disorders and primary care treatment.

Collection intervals

For preoperative PROMs, the ISAR PROMs Working Group recommends collecting PROMs within 3–4 weeks before surgery and not on the day of surgery. Recommendations for postoperative PROMs collection vary between hip and knee arthroplasty. For hip replacement patients, 1 year after surgery is the optimum follow–up time, but it may be better to measure knee replacement patients at 18 months to determine full capacity of recovery. However, for good comparability between arthroplasty registries worldwide, the Working Group recommends follow-up measures at 1 year after surgery for both hip and knee arthroplasty.

Frequency of response

There is a wide variation in frequency of response in PROMs collection between joint registries. The ISAR PROMs Working Group therefore attempted to advise on a desirable minimum frequency of response. There are no general guidelines available from research papers or journals about an acceptable frequency of response in reporting PROMs data. The instructions for authors of the Journal of the American Medical Association (2015) gives a requirement of ≥60% as a sufficient frequency of response. According to ISAR bylaws, full ISAR membership requires ≥80% completeness of procedures (International Society of Arthroplasty Registries 2015), a threshold that was selected arbitrarily and that reflected the bias associated with missing data. Nevertheless, the ISAR PROMs Working Group proposes a 60% threshold for an acceptable frequency of response because of external difficulties in capturing PROMs that may be unrelated to survey logistics. Registries that collect detailed demographic data may use imputation models to compute data for non-responders.

Definitions of frequency of response

Frequency of response is usually calculated annually, and the calculation may depend on whether PROMs are captured for all procedures or for a sample. For a given measurement interval in all patients, the total number of questionnaires returned or data entries, each matched to a registered procedure, is divided by the total number of procedures registered during the corresponding year:

For follow-up responses, the number of responses returned is divided by the difference between the total number of procedures registered during the corresponding year minus the number of joints in patients who died before follow-up (FU):

The number of procedures should be adjusted to the number intended for inclusion in the program. For logistic reasons, most registries do not collect preoperative PROMs for non-elective operations; therefore, non-elective procedures are usually excluded from calculations of response frequency.

Characterization of non-responders is recommended to ensure that data are not invalidated because of non-response bias. Furthermore, some patients may not respond before or after surgery, and the frequency of patients who have complete pre- and postoperative data in final analyses may be lower than the frequency of respondents before or after surgery. It is therefore important to maximize patient response both before and after surgery.

Reporting of missing values

The proportion of entries with incomplete or missing values should be reported per item. Measure-specific guidelines often provide guidance about how to address missing items in summary scores. Conditional mean single imputation is a common method to address missing items, but there may be other and better methods for imputation. Instrument-specific guidelines are commonly available on how to address answers that have misplaced marks or ≥2 options marked for the same item. These inaccuracies should be addressed according to measure-specific guidelines.

Establishment of a patient-reported outcome measures registry program

Establishing a PROMs program is a major challenge. It is helpful to establish a project team that includes a dedicated project leader, members of the registry management or executive committee, orthopedic surgeons who have large networks in the country or region, registry coordinators, and PROMs researchers.

Several established PROMs programs began as a small test project. It may be helpful to launch a small test project with a few providers to test feasibility and learn from early experiences before the full-scale registry program is established. The ISAR PROMs Working Group identified the importance of obtaining a critical minimum number of participating orthopedic surgeons or hospitals (representing about 10% of all arthroplasties performed) as a key factor for success in implementation. The numerous challenges in managing local logistics may require the support of an influential colleague locally. Hospital and clinic participation may be facilitated by early strategic registry staff visits to the site to inform, encourage, and assist at the start of PROMs collection. After the first participant hospitals show successful operational data collection, other hospitals may join. Some registries that have high response frequencies inform clinical sites about successful strategies to integrate capture of PROMs with preoperative clinic flow and about optimal postoperative direct-to-patient capture of PROMs.

We recommend minimization of the number of items included in a PROMs program, but the set of questions should reflect the essential constructs sought (primarily pain and physical function) and should be meaningful to patients. The ISAR PROMs Working Group recommends selection of 1 generic and 1 specific PROMs instrument, and it proposes that all PROMs programs should use 2 single-item measures (1 measure for joint pain and 1 measure for satisfaction with the results of surgery) to harmonize the measures.

Risk factors and case-mix adjustment

Case-mix variables

Case-mix adjustment is required to limit unjust comparisons of PROMs after joint replacement. Case-mix adjustment may facilitate comparison of outcomes between centers, providers, or countries by minimizing variability due to patient characteristics. However, the collection of case-mix variables in addition to PROMs may increase the burden on patients and healthcare providers. Thus, the choice of case-mix variables must be carefully balanced to obtain adequate frequency of response.

It may be difficult to determine whether—and how extensively or precisely—to include factors in case-mix adjustment models, such as diagnosis, previous interventions, body habitus, comorbidities, socioeconomic status, ethnicity, physical function, and personality traits. Both adjusted and unadjusted PROMs results should be reported, regardless of the case-mix variables used, to ensure transparency and comprehensive interpretation.

Variables to consider in case-mix adjustment

Well-designed randomized controlled trials attempt to minimize systematic differences between groups through randomization. Observational cohorts do not have this luxury. In an observational study, we need to minimize bias but accept that some bias may remain. A comprehensive set of confounding variables has to be defined, but it is most likely impossible to collect and adjust for all confounding variables. Numerous preoperative patient and clinical factors are associated with variation in postoperative pain, physical function, general health, and patient satisfaction (Table 2). Older age and female sex are associated with poorer postoperative function after total joint replacement (Jones et al. 2001, Franklin et al. 2008, Rolfson et al. 2011a, Judge et al. 2012b, Williams et al. 2013, Gordon et al. 2014a). Although higher body mass index may be associated with lower improvement in pain and function (Lübbeke et al. 2007, Franklin et al. 2008, Jameson et al. 2014b), the effect of it on outcome is small (Lübbeke et al. 2007, Jameson et al. 2014b), and body mass index may not be a meaningful predictor of outcome after total joint replacement (Judge et al. 2012b, 2014).

Table 2. Case-mix variables in arthroplasty registries^a

Category Case-mix variable	Level of data^b	Registries that directly record the variable^c	Registries that obtain variable the via linkage^d	References
Demographic
Age	1	All		Jones et al. 2001, Franklin et al. 2008, Rolfson et al. 2011a, Judge et al. 2012b, Williams et al. 2013, Gordon et al. 2014a, SooHoo et al. 2015.
Sex	1	All		Rolfson et al. 2011a, Judge et al. 2012b.
Dependency/marital status	2	FORCE-TJR, NJR	SHAR, SKAR	Judge et al. 2012, NHS England Analytical Team 2013, Greene et al. 2014a.
Work status	2	FORCE-TJR		Judge et al. 2010.
Socioeconomic status
Education	2	ACORN, FORCE-TJR, GAR, HSS, MARCQI	DHR, DKR, NAR, SHAR, SKAR	OECD 1999, UNESCO 2006, Judge et al. 2010, Greene et al. 2014a.
Deprivation index	2	RACat	NJR	Judge et al. 2012, Neuberger et al. 2012, 2013, NHS England Analytical Team 2013.
Joint-specific history
Diagnosis	1	All		Judge et al. 2012.
Previous surgery	2	ACORN, BAR, CJRR-Calif, DKR, FAR, FORCE-TJR, GAR, LROI, NAR, NJR, RIAP, RIPO, SKAR		NHS England Analytical Team 2013.
Body habitus
Height and weight (BMI)	2	ACORN, AJRR, AOANJRR, CJRR-Calif, CJRR-Canada, DKR, FORCE-TJR, GAR, HJR, HJRR, HSS, KP-TJRR, LROI, MARCQI, NJR, RACat, RIPO, SIRIS, SHAR, SKAR		Lübbeke et al. 2007, Franklin et al. 2008, Jameson et al. 2014b, Judge et al. 2014.
Lifestyle factors
Smoking	3	AJRR, FORCE-TJR, GAR, LROI, MARCQI, SHAR
Level of activity	3	CJRR-Calif, GAR, HJR, HSS
Alcohol abuse	3	GAR, HJR, MARCQI		Jenkins et al. 2011.
Comorbidities
Charlson Comorbidity Index	2	DHR, FORCE-TJR, HJRR, HSS	NAR, NJR, SHAR, SKAR	Bjorgul et al. 2010, NHS England Analytical Team 2013, Greene et al. 2015.
Elixhauser Comorbidity Index	2	KP-TJRR, RACat	NAR, MARCQI, SHAR, SKAR	Greene et al. 2015.
ASA class	2	ACORN, AJRR, AOANJRR, CJRR-Calif, GAR, HJRR, HSS, KP-TJRR, LROI, MARCQI, NAR, NJR, NZJRR, SHAR, SIRIS, SKAR		Bjorgul et al. 2010, Hooper et al. 2012, Judge et al. 2012, SooHoo et al. 2015.
Charnley class	2	DHR, FORCE-TJR, GAR, HJR, LROI, NAR, RIPO, SIRIS, SHAR, SKAR		Dunbar et al. 2004, Bjorgul et al. 2010, Rolfson et al. 2011a.
Back pain	2	FORCE-TJR		Ayers et al. 2013.
Mental health/depression	2	GAR, FORCE-TJR, HJR		Franklin et al. 2008, Rolfson et al. 2009, Judge et al. 2012.
Preoperative PROMs
Pain, function, and/or HRQoL	3	ACORN, AJRR, CJRR-Calif, FORCE-TJR, GAR, HJR, HSS, INOR, LAR, LROI, MARCQI, RIAP, SHAR, SKAR	NJR (through NHS)	Franklin et al. 2008, Rolfson et al. 2009, Judge et al. 2011, 2012, Greene et al. 2015, Maratt et al. 2015, SooHoo et al. 2015.

a Case-mix variable is a variable that is predictive of improvement in an outcome measure. Abbreviations: ASA, American Society of Anesthesiologists Physical Status Classification; BMI, body mass index; NHS, National Health Service England; HRQoL, health-related quality of life; PROMs, patient-reported outcome measures.

b Register level of data (Rolfson et al. 2011b). [Authors: not very clear/language editor]

c Registries: ACORN, Arthroplasty Clinical Outcomes Registry National Australia; AJRR, American Joint Replacement Registry; AOANJRR, Australian Orthopaedic Association National Joint Replacement Registry; BAR, Belgian Arthroplasty Register ORTHOpride; CJRR-Calif, California Joint Replacement Registry; CJRR-Canada, Canadian Joint Replacement Registry; DHR, Danish Hip Arthroplasty Register; DKR, Danish Knee Arthroplasty Register; FAR, Finnish Arthroplasty Register; FORCE-TJR, Function and Outcomes Research for Comparative Effectiveness in Total Joint Replacement; GAR, Geneva Arthroplasty Registry; HJR, Harris Joint Registry; HJRR, HealthEast Joint Replacement Registry; HSS, Hospital for Special Surgery Hip and Knee Joint Replacement Registry; INOR, Irish National Orthopaedic Register; KP-TJRR, Kaiser Permanente National Total Joint Replacement Registry; LAR, Lithuanian Arthroplasty Register; LROI, Dutch Arthroplasty Register; MARCQI, Michigan Arthroplasty Registry Collaborative Quality Initiative; NAR, Norwegian Arthroplasty Register; NJR, National Joint Registry for England, Wales, Northern Ireland and the Isle of Man; NZJR, New Zealand Joint Registry; RACat, Catalan Arthroplasty Register; RIAP, Italian Arthroplasty Registry Project; RIPO, Registro dell’implantologia Protesica Ortopedica (Register of the Orthopaedic Prosthetic Implants); SHAR, Swedish Hip Arthroplasty Register; SIRIS, Swiss Implant Register; SKAR, Swedish Knee Arthroplasty Register.

d Obtained by linkage to other health data registries.

Preoperative status is a strong predictor of pain and function after total joint replacement (Franklin et al. 2008). Patients who have more pain or poor function before surgery have a greater likelihood of greater postoperative gains (Franklin et al. 2008, Rolfson et al. 2009, Judge et al. 2012b, Greene et al. 2015). The ISAR PROMs Working Group recommends that registries that collect postoperative PROMs should also collect preoperative measures of pain and function to risk-adjust adequately when comparing outcomes between clinical groups. In addition, collection of pre- and postoperative PROMs may enable calculation of the degree of change or improvement after surgery.

Measures of socioeconomic status such as income, work status, and education affect postoperative PROMs (Table 2) (Greene et al. 2014a, Neuburger et al. 2012, 2013). However, these measures are recorded inconsistently in registries worldwide. Currently, United States Medicare is debating the use of socioeconomic status in case-mix adjustment.

The primary diagnosis or cause of knee or hip pain and disability varies between joint replacement patients, and diagnosis affects outcome (Mourão et al. 2009, Liao et al. 2015, Schrama et al. 2015). Although primary knee and hip osteoarthritis are the most common indications for knee or hip replacement patients, other acute and chronic conditions may result in surgery such as inflammatory conditions (rheumatoid arthritis), osteonecrosis, congenital disorders, femoral neck fracture, or sequelae following traumatic ligamentous injuries. There is no consensus about the optimal system to record the joint diagnosis (Bijlsma et al. 2011). Furthermore, some patients have undergone previous joint interventions of various kinds, such as arthroscopic procedures or osteotomy, and there is no consistent convention to document previous procedures. Research is needed to refine the categorization of joint pathology and previous procedures that may be important in case-mix adjustment. The ISAR PROMs Working Group recommends documentation of diagnosis such as primary osteoarthritis or other joint conditions.

Medical and musculoskeletal comorbidities are associated with varied patient-reported outcomes after joint replacement (Dieppe and Lohmander 2005). Administrative or billing data include lists of coexisting medical conditions that may be used for risk adjustment in summarized measures (such as Charlson comorbidity index, Elixhauser comorbidity measure, or number of comorbidities) or specific conditions (such as presence or absence of diabetes mellitus). Depending on the condition, these measures have the potential to improve prediction of postoperative pain and function. However, the role of medical risk factors may be small in understanding the variation in PROMs (Greene et al. 2015). Furthermore, administrative medical comorbidity coding may differ between centers and countries, and may be affected by the quality of data entry and reimbursement procedures. Although medical comorbidities are used regularly in case-mix adjustment, musculoskeletal comorbidities and risky behaviors such as smoking and alcohol use are less often included (Bijlsma et al. 2011). These comorbidity measures are not documented consistently in registries or administrative data (Greene et al. 2015). As recognized in the Charnley classification for grading of disabilities, outcomes from hip arthroplasty may be affected by the severity of pain in the contralateral hip, total osteoarthritis burden, and the presence of systemic disease that interferes with function, and these factors may also affect postoperative outcomes after knee arthroplasty (Jenkins et al. 2011, NHS England Analytical Team 2013, Greene et al. 2015). Although the Charnley classification has traditionally been assessed by the surgeon, a literature search did not show any study on the interobserver reliability of the Charnley classification. In the Swedish Hip and Knee Arthroplasty Registers, the classification is used as reported by the patient (Table 2) (Rolfson et al. 2011a, NHS England Analytical Team 2013). Coexisting contralateral knee osteoarthritis and low back pain have also been reported to affect pain and function after total joint replacement (Ayers et al. 2013). Moreover, the American Society of Anesthesiologists Physical Status Classification (ASA) is often documented in operative records and has been associated with PROMs after joint replacement (Dunbar et al. 2004).

In summary, medical and musculoskeletal comorbidities influence postoperative PROMs and they should be included in research and refined comparisons of outcomes between groups. However, inconsistent and incomplete documentation may limit the use of these comorbidity measures in routine postoperative risk-adjustment models. Overall health preoperatively (the sum of comorbidities) may be determined from patient-reported general health surveys such as EQ-5D, SF-12, or SF-36, and should be included as a factor in case-mix adjustment models (Browne et al. 2008). The quality of the case-mix variables depends on the completeness and accuracy of the data. The data may be collected from patients, physicians, or healthcare systems.

There are pitfalls associated with insufficient case-mix adjustment (residual confounding), differences in the way case-mix variables are assessed between centers, and the constant risk fallacy (Nicholl 2007) that occurs when the relation between the risk factor and outcome differs between groups. In addition, patient factors may interact with care provided and may differ between hospitals or countries.

The ISAR PROMs Working Group recommends that age, sex, diagnosis at joint, and preoperative health status (pain and function) score should be included in case-mix adjustment models to ensure valid comparisons of postoperative PROMs between clinical settings or countries. In addition, case-mix adjustment models should include preoperative PROMs, education level (as a measure of socioeconomic status), and Charnley classification (as a measure of comorbidity). Additional research is necessary to evaluate the usefulness and performance of other patient factors in case-mix adjustment models.

Interpretation and statistical analysis

Improvement or absolute level of pain, function, and health status

The aim of arthroplasty is to return patients to the highest possible level of pain relief, function, and health status. Patients who have severe symptoms before intervention have the greatest potential for improvement. However, patients who report the worst preoperative health states may not have as good an outcome after surgery than patients who report fewer problems before surgery (Vogl et al. 2014). Furthermore, the bounded design of instruments may limit differentiation between good and excellent outcomes. Evaluation of outcomes from the patient standpoint must therefore consider the amount of change as well as the absolute level of pain, function, and HRQoL at a specific time after surgery.

There is a strong positive association between pre- and postoperative PROMs scores. The magnitude of change in PROMs scores that occurs from before to after joint replacement surgery may cause incorrect conclusions about the quality of healthcare providers. It is quite probable that different surgeons would operate on patients who have different preoperative levels of severity, due in part to healthcare funding, surgeon training, and referral patterns. Arthroplasty surgeons and researchers are therefore encouraged to engage in the organization of the care of arthritis symptoms and arthroplasty. 

Missing data—categories and causes

The goal of arthroplasty PROMs analysis is to make valid inferences and learn from the data to improve arthroplasty care. Missing data can introduce bias. In prospective collection of registry data, there are often missing data and this cannot be entirely avoided. The cause of missing data must therefore be considered before starting the analysis.

Missing data can be classified into 3 categories (Little and Rubin 2002): (1) missing completely at random (MCAR), where the reason for missing data is unrelated to any outcome of interest; (2) missing at random (MAR), where the reason for missing data depends on known covariates; and (3) missing not at random (MNAR), where the reason for missing data is associated with unknown or unmeasured covariates.

Patients who have missing data may be systematically different from patients with complete information. It is important to understand the cause of the missing data and to differentiate between missing data due to item non-response (failure to complete an item) and missing data due to unit non-response (where the patient is lost to follow-up). Item non-response in PROMs is frequently observed and associated with difficulty, relevance, and importance of the question to the patient. In HOOS and KOOS, sports-domain and recreation-domain questions are the most commonly skipped items. When the outcome depends on a patient’s expectations in performing high-impact activites, the effect on these surveys may be biased unless the expectation in sporting activities is considered in the analysis. Unit non-response occurs when patients are lost to follow-up for administrative reasons or because of comorbidities, unwillingness to provide additional information, or death. Thus, variables associated with missing data should be identified and controlled for in the analysis.

The best method of addressing missing data is to minimize item and unit non-response during data collection. This can be achieved by making questionnaires as short as possible, ensuring that the PROMs are appropriate for and relevant to the population of interest, and using questions that are easily interpreted. Strategies that maximize frequency of response for collection of postoperative data may also be considered, such as envelope teasers, recorded delivery, monetary incentives, shorter questionnaires, pre-notification, and follow-up contact (Edwards et al. 2002, 2007).

Missing data—strategies

It is important to understand the consequences of missing data and to adopt an appropriate analytical strategy. When data are MCAR or MAR, complete case analysis (list-wise deletion) can be used to generate estimates with minimal bias, despite the inefficiency associated with calculation of standard errors. However, when attempting to recover inefficiency, other methods may be employed.

With MCAR or MAR assumptions, analyses such as multiple imputation and full information maximum likelihood (FIML) make allowances for missing data and recover inefficiencies associated with complete case analysis. In multiple imputation, realistic guesses are made about the missing value(s), and the uncertainty is propagated to the analyses to correct the standard errors. In FIML, parameters and standard errors are estimated in 1 step. Multiple imputation and FIML routines are available in most statistical software packages. Although multiple imputation is a general method that can be used with most estimation methods, FIML is used with structural equation modeling. However, modern structural equation modeling programs enable a wide class of generalized linear models and may be implemented easily.

Despite the wide availability of multiple imputation and FIML methods, several older single-imputation methods are frequently used but are not recommended. These include unconditional mean, conditional mean, and last observation carried forward (LOCF). These methods perform poorly, may introduce substantial bias, and underestimate standard errors. Current practices for handling of the problem of missing data have been reviewed (Graham 2009).

Analysis approaches and case-mix adjustment

Descriptive statistics should be reported—including mean, standard deviation, median, and interquartile range for continuous variables and frequency, proportion, and rate for categorical variables. Both preoperative and follow-up information should be summarized when a PROM is the primary outcome of interest. The initial state and progression of PROMs should be included in analyses. The role of confounding and case mix should be considered in PROMs data analysis, and adjustment for confounding and case-mix factors should be guided by causal knowledge (Hernán et al. 2002). Confounding and case-mix factors may be adjusted (1) in the design, with restriction or matching techniques such as propensity or radius, or (2) in the analysis, using inverse probability weighting, stratification, restriction, and multivariable regression.

The clinical relevance of changes in patient-reported outcome measures

A common question concerns the clinical relevance of changes in PROMs. Although every measure should be interpreted by defining thresholds for perceived improvement or deterioration, these thresholds may be difficult to define. Several methods are available for this purpose, but there is no consensus about the optimal method (King 2011). The minimal change or difference estimates calculated for a specific PROMs instrument may vary with the method used, intervention features, population characteristics, and instrument range. There is concern about universal thresholds of minimum important difference, and more research is needed on the application of such values in arthroplasty outcomes evaluation.

Statistical testing has been used to compare providers. However, it may be more informative to supply proportional data about patients improving or deteriorating, including well-grounded thresholds for minimum important differences.

Minimal clinically important difference and minimal detectable change

Clinicians and researchers rely on PROMs to assess and compare treatments and make decisions for clinical practice. However, numerical PROMs scores may lack direct clinically relevant meaning. Thus, minimal clinically important difference (MCID) and minimal detectable change (MDC) have been developed to represent the threshold needed to define treatment effectiveness (Rolfson et al. 2015). A PROM score that reaches MCID or MDC confirms clinical relevance and provides justification for implementation in clinical practice.

There is no standard MCID or MDC, because the MCID or MDC is specific to different PROMs, conditions, and populations. The MCID and MDC should be interpreted with caution and should consider measurement error for the PROM. This may be estimated with the MDC90, which is an MDC estimate that has a conventional confidence level of 90% (Kennedy et al. 2005, Kovacs et al. 2008). The interpretation of the MDC90 is that 90% of truly stable patients will show random variation of less than this magnitude when assessed on several occasions.

Involvement of statisticians and epidemiologists

Statistical analysis and interpretation of PROMs data is complex. The ISAR PROMs Working Group therefore recommends that registries should involve adequately trained biostatisticians and/or epidemiologists in processing and publishing PROMs data.

Future directions

Integration of patient-reported outcome measures into clinical decision-making

Discussions about risks and expected benefits of medical interventions are an integral component of the shared decision-making process in the encounter between healthcare professionals and patients (Makoul et al. 2006). These discussions provide patients with informed choices about treatment options that are based on the best evidence currently available (Lurie and Weinstein 2001). Although widely endorsed, the implementation of shared decision-making in clinical practice is limited, in part because of the lack of tools to display evidence and support the process (Weinstein et al. 2007). The decision to proceed with arthroplasty should be based on a most comprehensive benefit-risk assessment and on the patient’s preference. The benefit-risk tool should be derived from multiple measures including information about expected PROMs (Dieppe et al. 2011). PROMs data collected in registries will help to establish these decision-making tools, and early developments are in progress (Greene 2015).

Assessment of the usefulness of patient-reported out­come measures in post-marketing implant surveillance

Only a few registry-based studies have used PROMs in the comparison of different types of implants (Lübbeke et al. 2014) and surgical techniques (Jameson et al. 2014a, Lindgren et al. 2014). In the context of the urgently needed improvement of post-marketing implant surveillance, it is important to determine the value of PROMs as an early indicator of implant failure.

Computer-adaptive testing

The US National Institutes of Health (NIH) have initiated the Patient-Reported Outcomes Measurement Information System (PROMIS) to standardize PROMs used in studies that are funded by the NIH (Patient-Reported Outcomes Measurement Information System 2015). Some of the PROMIS measures use computer-adaptive testing (CAT) to minimize questionnaire burden on patients and to avoid floor and ceiling effects. Large banks of questions have been established for particular health status domains or constructs. The CAT survey pulls individual questions from the item bank. Based on the response to previous questions, the CAT system assigns the next question using item response theory, to expose a minimum set of relevant questions to the respondent and to create a summary score for the domain. The current feasibility of using CAT surveys with PROMs in arthroplasty registries is questionable, because CAT surveys are computer-based and many registries use paper-based forms. Furthermore, the content validity of CAT surveys has not been established for arthroplasty patients. Further research is needed on the validity of using CAT surveys with arthroplasty patients before adoption by registries and comparison with previously reported outcomes.

International comparisons

Harmonization of basic elements of PROMs collection and reporting in arthroplasty registries worldwide will enable international comparisons. There are international differences in culture, provision of healthcare, response patterns, priorities, and needs. International comparisons have the potential to identify successful practices. Although it is not feasible to suggest a universal standard for a complete set of relevant measures, agreement about a minimum set of PROMs and case-mix variables will facilitate international comparisons. Comprehensive measures may be compared by established crosswalk algorithms between different instruments.

Expansion of measures into the full continuum of care

Arthroplasty is a subset of care available for degenerative hip or knee conditions. It is desirable to monitor pain, function, and general health status for the continuum of care of degenerative joint disease before and after joint replacement surgery. Broader monitoring may improve our understanding about timing of surgery, arthroplasty indication, trajectories of patients who are not candidates for joint replacement, and factors associated with successful disease management.

The ISAR PROMs Working Group thanks Adrian Sayers for valuable contributions to this report. We also thank Elly Trepman for help in reorganizing the original internal report of the ISAR PROMs Working Group into 2 manuscripts.

All the authors participated in the conception of the study. OR, ALW, EB, PF, SL, and GD drafted the manuscript. All the authors were involved in critical revision of the article.