Preference Testing in Medical Devices: Current Framework and Regulatory Gaps

Figures & data

Table 1 Qualitative and Quantitative Preference Methods

A.Qualitative Preference Testing Methods
Method	Description	Example (Health Industry Related)
Delphi Method	Primarily used by groups of experts to gain consensus through controlled feedback when there is limited or insufficient knowledge/evidence on the topic of discussion.^Citation41,Citation52,Citation53	● Creation of guidelines and/or treatment protocols (eg creation of design guidance tools – Safety Risk Assessment for healthcare settings; The Center for Health Design, 2017).^Citation53 ● Consensus on recommendations for the use of fluorodeoxyglucose positron emission tomography (FDG-PET) in the detection of dementing neurodegenerative disorders.^Citation49
Dyadic Interview	Two-person interviews that combine the advantages of individual and focus groups interviews by assisting the researcher to formulate favorable circumstances for social interaction and depth.^Citation54	Photovoice dyadic interviews in people with dementia to evaluate the well-being of the patients (eg see studies from Wang and Burris, 1997).^Citation54
Concept Mapping	Concept mapping is a mixture of methods that involve brainstorming and unstructured sorting via multiple expert sources and is analyzed by multidimensional scaling and/or hierarchical cluster analysis statistical methods.^Citation55	Concept mapping can be used to guide a) action planning, b) program development, and c) evaluation procedures and/or measurements for the health sector:^Citation55 ● Development of conceptual frameworks in depression, anxiety, Quality of Life/Care (QoL/QoC). ● Development of guidelines in public health management (eg public health management of lower prevalence chronic conditions such as epilepsy). ● Stakeholder’s perspective identification in Traumatic Brain Injury (TBI), primary health care. ● Health planning/evaluation in mental health/illness, telemedicine in child protective services.
B.Quantitative Preference Testing Methods
Method	Description	Example (Health Industry Related)
Discrete Choice Experiment (DCE)	DCEs assess preferences and choices via examining various constituents of treatments, services, and trade-offs among involved stakeholders.^Citation24,Citation56	DCEs have been broadly accepted and used in pharmaco-economics and outcomes research that involve evaluation of patient preferences for treatments, drug comparison, biologics, and medical devices:^Citation19 ● Study of Patient Preferences Regarding Devices for Obesity, 2012 (results published in 2015). ● Study on the benefits and risks of rofecoxib relative to naproxen, 2010. ● Johnson & Johnson study on rivaroxaban for the prophylaxis of deep vein thrombosis (DVT) and pulmonary embolism (PE) in patients undergoing hip or knee replacement surgery, 2009.
Best-worst Scaling (BWS)	Types of Surveys to evaluate lists of attributes and/or profiles to signify the most or least appealing between them.^Citation40,Citation56	BWS can serve as a valuable tool to identify patients’/HCPs preferences for health related QoL:^Citation56,Citation57 ● Pilot BWS study as an empirical example conducted in 2005 to evaluate the QoL of elderly people (aged >65).
Q-Methodology	A rank-ordering method (agree-disagree) that associates the assets of both qualitative and quantitative methods and aims to appraise subject’s preferences based on a “forced normal distribution”.^Citation58	Q-Methodology may act as a beneficial apparatus to form the basis for examining attitudes (attributes that form cognitive relationships and subsequently behavior) within the fields of health education and promotion.^Citation58
Control Preference Scale (CPS)	CPS as a method utilizes the Unfolding statistical Theory (determination of preference distribution) and is applied to assess patient preferences regarding the desired control a patient aims to have over their medical treatment.^Citation40	CPS is broadly used in health care settings to evaluate the preferences of patients that deal with life-threatening diseases (eg breast cancer, prostate cancer, etc.):^Citation59 ● Henrikson et al, exploratory interview study among male patients suffering from localized prostate cancer, 2011.
Quantitative Preference Testing Methods
Method	Description	Example (Health Industry Related)
Standard Gamble (SG)	Estimation/measurement of cardinal preferences that are related to chronic or temporary health outcomes.^Citation60	SG provides the ability to measure multiple health states to generate testable comparisons among different populations and clinical settings:^Citation60 ● Yen et al, study on a Cost–Effectiveness Analysis of Treatment Strategies for Acute Uncomplicated Pyelonephritis in Women, 2003.^Citation61
Threshold Technique (TT)	A patient preference method that detects the most amount of change a patient will accept in one attribute to achieve a change in another attribute.^Citation40,Citation62	TT is broadly used in empirical health related studies to provide evidence on patient/HCP preferences and the accuracy of these preferences in terms of their results and estimates:^Citation62 ● Devereaux et al, prospective observational study on differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation, 2001 (TT used to evaluate patient/HCP preferences towards the reduction of stroke risk).^Citation63
Swing Weighting (SW)	Statistical preference method that evaluates attribute decision changes through the swinging effect of attributes and their assigned weights.^Citation40,Citation64	SW is used to obtain patient preferences based on treatment benefit-risk trade-offs (attributes included may be the clinical benefit, adverse events, convenience, etc.):^Citation64 ● Tervonen et al, empirical study on swing weighting and discrete choice experiments for elicitation of patient benefit‐risk preferences.
Paired Preference Test	Paired preference tests examine statistically significant preferences on specified sensory attributes among two options/products for a given population.^Citation43	Paired comparisons assist with identifying patient preferences in multiple clinical settings: ● Sutherland et al, study on measuring satisfaction with health care: a comparison of single with paired rating strategies, 1989.^Citation65

Table 2 Advantages and Disadvantages of Preference Testing¹

Advantages	Disadvantages
Preference tests are simple to set up.	Actual liking or disliking or both of the product in preference is unknown.
Respondents can easily understand the nature of the task.	Decision about “no preference” must be made.
Risk factors associated with preferences can be more precisely determined before test.	Paired comparison data are not particularly useful for product development because they give no real guidance on what is liked or disliked about a product (only which product is preferred).
Preference measures are likely to be seen as more relevant to consumers.
Preference is a criterion-free measure.

Note: Adapted from E2943-15, A. Standard guide for two-sample acceptance and preference testing with consumers; 2021; Available from: https://www.astm.org/e2943-15r21.html. Accessed June 16, 2022.¹

Table 3 FDA Recommended Qualities for Patient Preference Testing

Recommended Qualities	Description
1.Patient centeredness	The patient, and not the healthcare professional, should be the focus of the study. Risk-benefit preferences should be obtained from well-informed patients.
2.Representativeness of the Sample and Generalizability of Results	Preferences should be obtained from a representative sample of adequate size to enable generalisation of the results to the intended population. If preference differences between subgroups are of interest, the sample size of each subgroup should also be of adequate size.
3.Capturing Heterogeneity of Patients’ Preferences	Individual patient preferences will vary, no matter the commonalities that may exist between individuals. This variability in individual patient preference needs to be accounted for. Note: PPI can help identify subgroup preferences of risk-benefit trade-offs (eg, patients with higher pain or increased severity of a condition/disease may be willing to accept more risk than those with a lower pain/less severity).
4.Established Good Research Practices by Recognized Professional Organizations	Guidelines for good research practices from recognized professional organizations should be used, such as the guideline from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR).^Citation66
5.Effective Communication of Benefit, Harm, Risk, and Uncertainty	The context of the benefit-risk trade-offs, the level of effectiveness and the severity of treatment-related harms need to be defined and patients need to understand probabilities using appropriate numeric, verbal, and graphic representations of uncertainty.
6.Minimal Cognitive Bias	Potential cognitive bias, such as framing, anchoring, simplifying heuristics, and ordering effect, should be minimized.
7.Logical Soundness	Internal-validity tests of logic and consistency should be included.
8.Relevance	Preferences in relation to harm, risk, benefit, and uncertainty should be obtained or well justified if not. Specific endpoints should be clearly defined in relation to their clinical outcomes.
9.Robustness of Analysis of Results	Appropriate analysis of the results is needed for correct interpretation of the data. Sources of uncertainty need to be understood and can be reported via confidence intervals and standard errors.
10.Study Conduct	The study should be conducted by trained research staff. If the study is self-administrated, the patients should be given a tutorial and quiz on how to complete the study.
11.Comprehension by Study Participants	Patients should fully comprehend the harm, risk, benefit, uncertainty, and any other medical information presented to them (ie, all questions must be fully understood for data to be valid).

Notes: Data from Food and Drug Administration. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-preference-information-voluntary-submission-review-premarket-approval-applications.¹⁰

Figure 1 The potential implication of preference testing within risk management.

Figure 2 Stage-gate design concept for medical devices, including the conduction of preference testing. Data from sources.^22,51

Figure 3 Usability testing in relation to patient/physician preferences.

Boulkedid R, Abdoul H, Loustau M, et al. Using and reporting the delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. doi:10.1371/journal.pone.0020476

 PubMed Web of Science ®Google Scholar

Niederberger M, Spranger J. Delphi technique in health sciences: a map. Public Health Front. 2020;8:457. doi:10.3389/fpubh.2020.00457

 Google Scholar

Taylor E. We agree, don’t we? The Delphi method for health environments research. HERD. 2020;13(1):11–23. doi:10.1177/1937586719887709

 PubMedGoogle Scholar

Nobili F, Arbizu J, Bouwman F, et al. European Association of Nuclear Medicine and European Academy of Neurology recommendations for the use of brain 18 F-fluorodeoxyglucose positron emission tomography in neurodegenerative cognitive impairment and dementia: delphi consensus. Eur J Neurol. 2018;25(10):1201–1217. doi:10.1111/ene.13728

 PubMed Web of Science ®Google Scholar

Morgan DL, Ataie J, Carder P, et al. Introducing dyadic interviews as a method for collecting qualitative data. Qual Health Res. 2013;23(9):1276–1284. doi:10.1177/1049732313501889

 PubMed Web of Science ®Google Scholar

Trochim W, Kane M. Concept mapping: an introduction to structured conceptualization in health care. Int J Qual Health Care. 2005;17(3):187–191. doi:10.1093/intqhc/mzi038

 PubMed Web of Science ®Google Scholar

Harrison M, Milbers K, Hudson M, et al. Do patients and health care providers have discordant preferences about which aspects of treatments matter most? Evidence from a systematic review of discrete choice experiments. BMJ Open. 2017;7(5):e014719. doi:10.1136/bmjopen-2016-014719

 PubMed Web of Science ®Google Scholar

Flynn TN, Louviere JJ, Peters TJ, et al. Best–worst scaling: what it can do for health care research and how to do it. J Health Econ. 2007;26(1):171–189. doi:10.1016/j.jhealeco.2006.04.002

 PubMed Web of Science ®Google Scholar

Consortium, M.D.I., A framework for incorporating information on patient preferences regarding benefit and risk into regulatory assessments of new medical technology; 2015.

 Google Scholar

Soekhai V, Whichello C, Levitan B, et al. Methods for exploring and eliciting patient preferences in the medical product lifecycle: a literature review. Drug Discov Today. 2019;24(7):1324–1331. doi:10.1016/j.drudis.2019.05.001

 PubMed Web of Science ®Google Scholar

Flynn TN. Valuing citizen and patient preferences in health: recent developments in three types of best-worst scaling. Expert Rev Pharmacoecon Outcomes Res. 2010;10(3):259–267. doi:10.1586/erp.10.29

 PubMed Web of Science ®Google Scholar

Cross RM. Exploring attitudes: the case for Q methodology. Health Educ Res. 2005;20(2):206–213. doi:10.1093/her/cyg121

 PubMed Web of Science ®Google Scholar

Henrikson NB, Davison BJ, Berry DL. Measuring decisional control preferences in men newly diagnosed with prostate cancer. J Psychosoc Oncol. 2011;29(6):606–618. doi:10.1080/07347332.2011.615383

 PubMed Web of Science ®Google Scholar

Alex G, Wyrwich KW. Wyrwich, health utility measures and the standard gamble. Acad Emerg Med. 2003;10(4):360–363. doi:10.1111/j.1553-2712.2003.tb01349.x

 PubMed Web of Science ®Google Scholar

Yen Z-S, Davis MA, Chen S-C, et al. A cost–effectiveness analysis of treatment strategies for acute uncomplicated pyelonephritis in women. Acad Emerg Med. 2003;10(4):309–314. doi:10.1111/j.1553-2712.2003.tb01341.x

 PubMed Web of Science ®Google Scholar

Hauber B, Coulter J. Using the threshold technique to elicit patient preferences: an introduction to the method and an overview of existing empirical applications. Appl Health Econ Health Policy. 2020;18(1):31–46. doi:10.1007/s40258-019-00521-3

 PubMed Web of Science ®Google Scholar

Devereaux PJ, Anderson DR, Gardner MJ, et al. Differences between perspectives of physicians and patients on anticoagulation in patients with atrial fibrillation: observational study. BMJ. 2001;323(7323):1218–1222. doi:10.1136/bmj.323.7323.1218

 PubMed Web of Science ®Google Scholar

Tervonen T, Gelhorn H, Sri Bhashyam S, et al. MCDA swing weighting and discrete choice experiments for elicitation of patient benefit-risk preferences: a critical assessment. Pharmacoepidemiol Drug Saf. 2017;26(12):1483–1491. doi:10.1002/pds.4255

 PubMed Web of Science ®Google Scholar

E2263-12, A. Standard test method for paired preference test; 2018. Available from: www.astm.org. Accessed June 16, 2022.

 Google Scholar

Sutherland HJ, Lockwood GA, Minkin S, et al. Measuring satisfaction with health care: a comparison of single with paired rating strategies. Soc Sci Med. 1989;28(1):53–58. doi:10.1016/0277-9536(89)90306-7

 PubMed Web of Science ®Google Scholar

E2943-15, A. Standard guide for two-sample acceptance and preference testing with consumers; 2021; Available from: https://www.astm.org/e2943-15r21.html. Accessed June 16, 2022.

 Google Scholar

Prosser LA. Statistical methods for the analysis of discrete-choice experiments: a report of the ISPOR conjoint analysis good research practices task force. Value Health. 2016;19(4):298–299. doi:10.1016/j.jval.2016.05.002

 PubMed Web of Science ®Google Scholar

FDA. Patient preference information - voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling; 2016. Available from: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-preference-information-voluntary-submission-review-premarket-approval-applications. Accessed June 16, 2022.

 Google Scholar

Pietzsch JB, Shluzas LA, Paté-Cornell ME, et al. Stage-gate process for the development of medical devices. J Med Device. 2009;3(2). doi:10.1115/1.3148836

 Google Scholar

King PH, Fries RC. Design of Biomedical Devices and Systems. Second ed. Crc Press; 2008.

 Google Scholar