Abstract
Background
The selection of important outcomes is a crucial decision for clinical research and health technology assessment (HTA), and there is ongoing debate about which stakeholders should be involved. Hemodialysis is a complex treatment for chronic kidney disease (CKD) and affects many outcomes. Apart from obvious outcomes, such as mortality, morbidity and health-related quality of life (HRQoL), others such as, concerning daily living or health care provision, may also be important. The aim of our study was to analyze to what extent the preferences for patient-relevant outcomes differed between various stakeholders. We compared preferences of stakeholders normally or occasionally involved in outcome prioritization (patients from a self-help group, clinicians and HTA authors) with those of a large reference group of patients.
Participants and methods
The reference group consisted of 4,518 CKD patients investigated previously. We additionally recruited CKD patients via a regional self-help group, nephrologists via an online search and HTA authors via an expert database or personal contacts. All groups assessed the relative importance of the 23 outcomes by means of a discrete visual analog scale. We used descriptive statistics to rank outcomes and compare the results between groups.
Results
We received completed questionnaires from 49 self-help group patients, 19 nephrologists and 18 HTA authors. Only the following 3 outcomes were ranked within the top 7 outcomes by all 4 groups: safety, HRQoL and emotional state. The ratings by the self-help group were generally more concordant with the reference group ratings than those by nephrologists, while HTA authors showed the least concordance.
Conclusion
Preferences of CKD patients from a self-help group, nephrologists and HTA authors differ to a varying extent from those of a large reference group of patients with CKD. The preferences of all stakeholders should form the basis of a transparent approach so as to generate a valid list of important outcomes.
Background
For some years now, more and more emphasis has been placed on patient-centered health care as well as on active patient involvement in clinical research and health technology assessment (HTA).Citation1–Citation4
The Institute of Medicine (IOM) defined patient-centered care as “Providing care that is respectful of and responsive to individual patient preferences, needs, and values, and ensuring that patient values guide all clinical decisions.”Citation5
The need for patient engagement in health care research is widely recognized, and measures are being undertaken in many areas to promote further engagement.Citation3,Citation6 One area of interest is the definition and rating of patient-relevant outcomes. However, the decision as to which outcomes are important enough to be evaluated in clinical research or HTA is still largely made by clinicians or HTA authors; patients are only occasionally involved in this process.
It is unclear to what extent the importance that clinicians or HTA authors assign to certain outcomes reflects the importance assigned by patients. Several studies have found discrepancies between preferences of patients and those of other stakeholders.Citation7,Citation8
Moreover, if patients are involved, this normally happens in an unstructured manner by consultation with a small group of patient representatives (eg, members of a self-help group). It is unclear to what extent they represent the perspective of the total population of patients with a specific condition.
Hemodialysis is a time-consuming and long-term intervention for the treatment of chronic kidney disease (CKD) and affects many different outcomes. While there are obvious ones, such as mortality, morbidity and health-related quality of life (HRQoL), others related to the impact on daily living or the provision of health care (eg, journey time to dialysis or its duration) may also be important.Citation9
We had previously conducted a survey of 4518 CKD patients treated by hemodialysis to investigate the relative importance that patients assign to various outcomes and found that the 3 most important ones were the safety of treatment, HRQoL and satisfaction with care.Citation10 However, the involvement of such a large patient sample in outcome prioritization in clinical research or HTA would not normally be feasible in practice.
We therefore aimed to investigate to what extent the preferences of 3 stakeholder groups normally or occasionally involved in outcome prioritization in clinical research or HTA deviate from the preferences of a large reference group of patients. For this purpose, we compared the preferences of members of a self-help group for CKD, nephrologists and HTA authors with those of a reference group with CKD in respect of the importance of hemodialysis-related outcomes.
Participants and methods
Stakeholder groups
Reference group: CKD patients from a quality management program in nephrology
The reference group consisted of CKD patients treated by hemodialysis in the centers of the Board of Trustees for Dialysis and Renal Transplant (Kuratorium fuer Dialyse und Nierentransplantation [KfH]), the largest nonprofit provider of hemodialysis in Germany. It comprises >200 units treating ~18,000 patients per year and sends an annual survey to all of its patients as a part of its quality management program “Quality in Nephrology” (QiN).Citation11 Our questionnaire was included in the survey of 2008. A detailed description has been published elsewhere.Citation10
CKD patients from a self-help group
As patient representatives, we included patients recruited from a regional self-help group for CKD, the Lower-Rhine Section of the Kidney Interest Group (Sektion Niederrhein, Interessengemeinschaft Niere e.V.). A paper-and-pencil version of the questionnaire was distributed during a regional meeting of this group; in addition, the group’s office was provided with paper-and-pencil versions to distribute in further meetings. The office also sent the link of the online version of the questionnaire to all members of the self-help group with known email addresses.
Nephrologists
We identified nephrologists by selecting the largest city of each of the 16 German federal states and identified outpatient dialysis centers in these cities via a Google search. We contacted a total of 80 centers that had a website with an email address. We sent them a link of the online questionnaire and asked them to participate and also to forward the link to further nephrologists. In addition, a nephrologist collaborating with our project disseminated the link in national nephrology meetings.
HTA authors
A total of 25 HTA authors included in the Institute for Quality and Efficiency in Healthcare database of external experts or personally known to the authors of this paper were contacted via email and provided with a link of the online questionnaire. They were asked to participate and also to forward the link to further HTA authors.
Questionnaire
Outcomes for inclusion in the questionnaire were identified using patient input; details are described elsewhere.Citation10 In short, a group discussion was held with patients from a self-help group to identify the important outcomes. A pretest was conducted with 5 patients, not otherwise involved in the project, to assess the comprehensibility of the questions and the completeness of the outcomes used.
Subsequently, 23 hemodialysis-related outcomes were identified and added to a routinely conducted annual survey (“Reference group: CKD patients from a quality management program in nephrology” section). We only considered patient-relevant outcomes, defined as how a patient feels, functions or survives.Citation12 Surrogate outcomes, that is, outcomes not directly perceptible by patients (eg, laboratory values), were not considered.
On the basis of a systematic literature search for instruments for assessing preferences, we chose a rating scale in the form of a discrete visual analog scale (VAS) to rate the preferences for the 23 outcomes.Citation7 This scale consisted of 9 categories, ranging from “not important” (1) to “very important” (9). A complete list of the 23 outcomes and their definition in the questionnaire is listed in . Patients from the reference group and the self-help group were asked to indicate how important the outcomes were to them by assigning a score from 1 to 9; similarly, nephrologists and HTA authors were asked to indicate how important they thought that these outcomes were to patients.
Table 1 Outcomes included in the questionnaire
Sociodemographic data were collected from all 4 groups. In addition, we collected information on diabetes diagnosis, time on dialysis in years, and education, and occupation, from the patients. From nephrologists, we collected information on work experience, the average number of patients treated and the work setting (hospital or practice).
Patients, nephrologists and HTA authors were informed about the purpose of the questionnaire and were asked to give informed consent. The paper-and-pencil versions were handed out together with a consent form and a return envelope. In the online version, the participants were requested to check a box to confirm their informed consent before proceeding with the questionnaire. The data of the entire questionnaire were collected within a quality assurance framework, and a specially assigned data protection commissioner ensured that patient data were dealt with correctly according to German data protection laws. Only data of patients who gave written informed consent were considered.
Data analysis
Descriptive statistics (mean scores and standard deviation) were used to calculate the ratings of the importance of the 23 outcomes. The results from the QiN surveyCitation10 were used as a reference to which the other groups were compared (reference group). Comparisons with this group were performed in the following 2 different ways:
In the first analysis, we compared the 7 top-ranked outcomes. The number of outcomes evaluated and reported is usually restricted both in clinical research and in evidence synthesis. While there is no general international consensus on the exact number of outcomes to be reported, the Cochrane Collaboration and the Grading of Recommendations Assessment, Development and Evaluation working group report up to 7 outcomes.Citation13,Citation14 Outcomes not included in this list are not usually considered in the weighing of benefits and harms of an intervention.
In the second analysis, we compared the extent of deviations in the rankings when all 23 outcomes were considered. We compared the rankings of the 3 stakeholder groups for each outcome with those of the reference group, counting how often the outcomes differed by 0–2, 3–5, 6–8 or >8 ranks. In addition, we compared the 7 top-ranked outcomes of the 3 stakeholder groups with each other, without considering the reference group.
Results
Participants
In the QiN study, we had previously analyzed 4518 questionnaires from hemodialysis patients. In this study, we received 49 completed questionnaires from the self-help group (). While most characteristics were similar in the 2 patient groups, patients from the self-help group were on average younger, had diabetes less often and had spent more time on dialysis. They were also more often employed and were more likely to have higher education. We received 18 completed questionnaires from HTA authors and 19 from nephrologists (). On average, HTA authors had 6 years of HTA experience and nephrologists had 20 years of experience in treating patients with CKD and, at the time of the survey, were each responsible for 40–150 patients.
Table 2 Characteristics of patient groups
Table 3 Characteristics of HTA authors and nephrologists
Ranking of outcomes
Comparison of the 3 stakeholder groups with the reference group
Variations in the 7 top-ranked outcomes
Detailed results of the importance ratings (mean and standard deviation) are listed in . shows the 7 top-ranked outcomes of all 4 groups. The following 3 outcomes were ranked within the top 7 by all groups: safety, HRQoL and emotional state.
Figure 1 Comparison of outcomes.
Abbreviations: GI, gastrointestinal; HRQoL, health-related quality of life; HTA, health technology assessment.
Self-help group versus reference group: 5 outcomes were ranked within the top 7 outcomes by both groups. Both groups ranked safety in the first place. However, the self-help group rated physical functioning (rank 2 versus rank 11) and traumatic punctures (rank 5 versus rank 9) as being more important than did the reference group. In contrast, hospital stays and accompanying symptoms were rated by the reference group as being more important.
Nephrologists versus reference group: 4 outcomes were ranked within the top 7 outcomes by both groups. Both groups had a similarly high ranking for satisfaction with care (rank 2 versus rank 3). However, nephrologists rated access to nursing staff (rank 1 versus rank 8), physical functioning (rank 5 versus rank 11) and nausea/drop in blood pressure (rank 6 versus rank 10) as being more important than did the reference group. In contrast, hospital stays, accompanying symptoms and hemodialysis duration were rated by the reference group as being more important.
HTA authors versus reference group: 4 outcomes were ranked within the top 7 outcomes by both groups. Both groups had a similarly high ranking for HRQoL (rank 1 versus rank 2). However, HTA authors rated work (rank 2 versus rank 23), life expectancy (rank 3 versus rank 15) and physical functioning (rank 5 versus rank 11) as being more important than did the reference group. In contrast, satisfaction with care, accompanying symptoms and hemodialysis duration were rated by the reference group as being more important.
Deviations when all 23 outcomes are considered
shows the extent of deviations in the rankings of all outcomes for each of the 3 stakeholder groups compared to the ranking by the reference group. In the comparison of the rankings by the self-help group with those by the reference group, 18 outcomes were ranked with very small or small differences and 5 outcomes were ranked with large or very large differences.
Figure 2 Extent of differences in outcome rankings between the self-help group, nephrologists and HTA authors versus the reference group.
Abbreviation: HTA, health technology assessment.
In the comparison of the rankings by the nephrologists with those by the reference group, 17 outcomes were ranked with small or very small differences and 6 outcomes were ranked with large or very large differences.
In the comparison of the rankings by the HTA authors with those by the reference group, 13 outcomes were ranked with small or very small differences and 10 outcomes were ranked with large or very large differences (even a difference of 21 ranks for 1 outcome, work).
Comparison of the self-help group, nephrologists and HTA authors
shows the 7 top-ranked outcomes of the 3 stakeholder groups. As stated, they agreed on 4 outcomes (safety, physical functioning, HRQoL and emotional state). Outcomes that were ranked within the top 7 solely by one group were traumatic punctures and hemodialysis duration (self-help group), access to nursing staff and nausea/drop in blood pressure (nephrologists), as well as work, life expectancy and hospital stays (HTA authors).
Figure 3 The 7 top-ranked outcomes of stakeholder groups.
Abbreviations: HRQoL, health-related quality of life; HTA, health technology assessment.
Discussion
The aim of this study was to investigate to what extent the preferences of 3 stakeholder groups (CKD patients from a self-help group, nephrologists and HTA authors) normally or occasionally involved in outcome prioritization in clinical research and HTA deviate from the preferences of a large reference group of patients with CKD. Only 3 outcomes, which were safety, HRQoL and emotional state, were consistently ranked within the top 7 outcomes by all 4 groups. Depending on the group asked, the outcomes included in the top 7 varied considerably. This is problematic as often only a single stakeholder group is involved in outcome prioritization in clinical research and HTA.Citation15,Citation16
Although the rankings of the self-help group were more similar to those of the reference group than the rankings of clinicians and HTA authors, there were notable deviations, which may be explained by the fact that survey results are affected by sampling bias if participants are selected nonrandomly,Citation17–Citation19 as was the case with the self-help group in our study, who, for instance, were younger than patients in the reference group. However, an exploratory analysis comparing patients of both groups within similar age strata did not yield results that were more similar to each other; factors other than age, therefore, seem to have led to different results. Specific factors that differ from those in the general patient population might be difficult to identify and even more difficult to control for. This has to be taken into account when considering preferences elicited only from patients in a specific subgroup.
The 2 outcomes ranked highest by the nephrologists were access to nursing staff and satisfaction with care at the dialysis clinic, indicating that clinicians place more emphasis on outcomes related to clinic organization than do patients and HTA authors.
Outcomes ranked quite differently by HTA authors than by the reference patients include life expectancy and the possibility to work despite dialysis; these were rated to be far more important by HTA authors. The fact that life expectancy (rank 15) was not considered by the reference patients to be as important as other outcomes might be explained by the fact that CKD is no longer an acute life-threatening condition. The possibility to work (ranked last) is understandably not so important for a patient group that is largely retired. When evaluating the importance of outcomes, the HTA authors were probably influenced by their past experiences concerning which outcomes can be reliably measured in clinical studies.
Strengths and limitations
The strength of our study is the possibility to compare the importance of outcomes for 3 stakeholder groups normally or occasionally involved in the process of outcome prioritization with the results of a large reference group of patients whose preferences are not usually available.
We chose a discrete VAS to enable the elicitation of preferences for a large number of outcomes. Other methods, such as conjoint analysis, analytical hierarchy process and utility assessment methods, are also used for this purpose, but as they are not suitable for dealing with the large number of outcomes analyzed in our study, they were not feasible alternatives.Citation7,Citation15 Although a ceiling effect was observed, we were able to establish a ranking of outcomes and to compare the ranks between the different samples.
Nonrepresentative samples (such as patients from a self-help group) do not necessarily provide exact estimates of the preferences of a large patient population. However, in decision-making in clinical research and HTA, the input of health care professionals is not based on a representative sample either.
Comparison with previous research
Studies using preference elicitation methods have been published increasingly over the last decade.Citation7,Citation16,Citation20 In our previous descriptive review on methods to prioritize outcomes, we identified several studies comparing preferences for the outcomes of different stakeholders; most of them compared preferences of patients and clinicians.Citation7 The studies identified were conducted in different health care settings and used different methods to evaluate the importance of outcomes. Most studies showed notable differences between preferences of patients and other stakeholders, that is, either different hierarchies or at least different strengths of preferences for outcomes.
We did not identify a single study comparing the preferences of any of the 3 stakeholder groups, and investigated with such a large reference group as the one included in our analysis; furthermore, we are not aware of studies comparing the preferences of HTA authors or dialysis patients with those of other groups.
Preference elicitation methods, especially multicriteria decision analysis (MCDA), are being increasingly used for reimbursement decisions in HTA.Citation20,Citation21 A systematic review of studies applying MCDA methods in decisions addressing the trade-off between costs and benefits found that the majority of studies used health outcomes as a criterion to elicit preferences from different stakeholder groups. Interestingly, health policy decision makers were more intensely involved, while patient involvement was limited.Citation15
Implications of research findings
The choice of important outcomes has implications for the evaluation of efficacy in clinical research and consequently of comparative effectiveness in HTAs. The explicit elicitation of preferences from different stakeholder groups can make the process of clinical research and HTA more transparent and add important dimensions.Citation22,Citation23
The question arises as to how outcomes reflecting patient preferences can be identified for decision-making in clinical research and HTA in a valid and feasible manner. Our study indicates that even though some deviations exist, the preferences of a self-help group might represent an adequate proxy for a large patient population with the same condition.
The preferences of clinicians and HTA authors deviated to a greater extent from those of the reference group but not so much that they should be discarded in cases where direct access to patient groups is not feasible. More importantly, their insights are relevant for outcomes not perceptible but still potentially important to patients (eg, clinical outcomes). Furthermore, they can advise on outcomes that might not be of major importance to the individual patient but to the health care system or society in general.
The Standardized Outcomes in Nephrology-Hemodialysis (SONG-HD) initiative is working on a core outcome set for all trials in hemodialysis involving patients and health professionals.Citation24 However, further research is needed to analyze how preferences of different groups can be implemented in research decisions.Citation25
Conclusion
Our study indicates that preferences of different stakeholders (CKD patients from a self-help group, nephrologists and HTA authors) differ to a varying extent from those of a large reference group of patients with CKD. The involvement of self-help groups might be a feasible means of including patient preferences in outcome prioritization, but the effects of the mode of sampling have to be considered in the interpretation of results. In addition, the preferences of health care professionals such as clinicians and HTA authors should be considered, as only the combination of all preferences forms the basis of a transparent discussion on important outcomes in clinical research and HTA.
Acknowledgments
The authors thank Natalie McGauran for medical writing support. We acknowledge support for the Article Processing Charge by the Deutsche Forschungsgemeinschaft and the Open Access Publication Fund of Bielefeld University.
Supplementary material
Table S1 Ratings of importance of outcomes by the stakeholder groups
Disclosure
The authors report no conflicts of interest in this work.
References
- FaceyKBoivinAGraciaJPatients’ perspectives in health technology assessment: a route to robust evidence and fair deliberationInt J Technol Assess Health Care201026333434020584364
- FaceyKMHansenHPPatient-focused HTAsInt J Technol Assess Health Care201127427327422004766
- DomecqJPPrutskyGElraiyahTPatient engagement in research: a systematic reviewBMC Health Serv Res2014148924568690
- MuhlbacherACJuhnkeCPatienten- und Bürgerpartizipation in der Entscheidungsfindung im Gesundheitswesen insbesondere bei der Bewertung von Arzneimitteln [Involving patients, the insured and the general public in healthcare decision making]Z Evid Fortbild Qual Gesundhwes2016110–1113644 German
- Institute of Medicine (US)Committee on Quality of Health Care in AmericaCrossing the Quality Chasm: A New Health System for the 21st CenturyWashington, DCNational Academy Press2001
- RyanMKinghornPEntwistleVAFrancisJJValuing patients’ experiences of healthcare processes: towards broader applications of existing methodsSoc Sci Med201410619420324568844
- JanssenIMGerhardusASchroer-GuntherMAScheiblerFA descriptive review on methods to prioritize outcomes in a health care contextHealth Expect20151861873189325156207
- MuhlbacherACJuhnkeCPatient preferences versus physicians’ judgement: does it make a difference in healthcare decision making?Appl Health Econ Health Policy201311316318023529716
- Urquhart-SecordRCraigJCHemmelgarnBPatient and caregiver priorities for outcomes in hemodialysis: an international nominal group technique studyAm J Kidney Dis201668344445426968042
- JanssenIMGerhardusAvon GersdorffGDPreferences of patients undergoing hemodialysis – results from a questionnaire-based study with 4,518 patientsPatient Prefer Adherence2015984785526170634
- StoffelMPBarthCLauterbachKWBaldamusCAEvidence-based medical quality management in dialysis – part I: routine implementation of QiN, a German quality management systemClin Nephrol200462320821815481853
- Biomarkers Definitions Working GBiomarkers and surrogate endpoints: preferred definitions and conceptual frameworkClin Pharmacol Ther2001693899511240971
- HigginsJGreenS homepage on the InternetCochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration2011 Available from: www.cochrane-handbook.orgAccessed November 9, 2016
- SchünemannHBrożekJOxmanAGRADE Handbook for Grading Quality of Evidence and Strength of Recommendation. Version 3.2 [updated March 2009]The GRADE Working Group2009 Available from: http://www.cc-ims.net/gradepro
- WahlsterPGoetghebeurMKrizaCNiederlanderCKolominsky-RabasPNational Leading-Edge Cluster Medical Technologies ‘Medical Valley EMNBalancing costs and benefits at different stages of medical innovation: a systematic review of Multi-criteria decision analysis (MCDA)BMC Health Serv Res20151526226152122
- GargonEGurungBMedleyNChoosing important health outcomes for comparative effectiveness research: a systematic reviewPLoS One201496e9911124932522
- KalsbeekWHeissGBuilding bridges between populations and samples in epidemiological studiesAnnu Rev Public Health20002114716910884950
- TyrerSHeymanBSampling in epidemiological research: issues, hazards and pitfallsBJPsych Bull2016402576027087985
- NeumarkDEStommelMGivenCWGivenBAResearch design and subject characteristics predicting nonparticipation in a panel survey of older families with cancerNurs Res200150636336811725938
- MarshKIJMThokalaPMultiple criteria decision analysis for health care decision making-emerging good practices: report 2 of the ISPOR MCDA emerging good practices task forceValue Health201619212513727021745
- AngelisAKanavosPValue-based assessment of new medical technologies: towards a robust methodological framework for the application of multiple criteria decision analysis in the context of health technology assessmentPharmacoeconomics201634543544626739955
- GagnonMPDesmartisMLepage-SavaryDIntroducing patients’ and the public’s perspectives to health technology assessment: a systematic review of international experiencesInt J Technol Assess Health Care2011271314221262085
- HaileyDWerkoSBakriRInvolvement of consumers in health technology assessment activities by Inahta agenciesInt J Technol Assess Health Care2013291798323217279
- TongAMannsBHemmelgarnBEstablishing core outcome domains in hemodialysis: report of the Standardized Outcomes in Nephrology-Hemodialysis (SONG-HD) Consensus WorkshopAm J Kidney DisIn press2016
- GagnonMPDesmartisMGagnonJFramework for user involvement in health technology assessment at the local level: views of health managers, user representatives, and cliniciansInt J Technol Assess Health Care2015311–2687725952585