Tactical Considerations for Designing Real-World Studies: Fit-for-Purpose Designs That Bridge Research and Practice

References

• US Food and Drug Administration. Framework for FDA’s real-world evidence program; 2018. Available from: https://www.fda.gov/media/120060/download. Accessed September 20, 2023.
   Google Scholar
• Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-world evidence — what is it and what can it tell us? N Engl J Med. 2016;375(23):2293–2297. doi:10.1056/NEJMsb1609216
   PubMed Web of Science ®Google Scholar
• Brass EP. The gap between clinical trials and clinical practice: the use of pragmatic clinical trials to inform regulatory decision making. Clin Pharmacol Ther. 2010;87(3):351–355. doi:10.1038/clpt.2009.218
   PubMed Web of Science ®Google Scholar
• Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464–475. doi:10.1016/j.jclinepi.2008.12.011
   PubMed Web of Science ®Google Scholar
• Ford I, Norrie J, Drazen JM. Pragmatic Trials. N Engl J Med. 2016;375(5):454–463. doi:10.1056/NEJMra1510059
   PubMed Web of Science ®Google Scholar
• Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20(8):637–648. doi:10.1016/0021-9681(67)90041-0
   PubMedGoogle Scholar
• Prentice RL. Surrogate endpoints in clinical trials: definition and operational criteria. Stat Med. 1989;8(4):431–440. doi:10.1002/sim.4780080407
   PubMed Web of Science ®Google Scholar
• US Food and Drug Administration. Diversity plans to improve enrollment of participants from underrepresented racial and ethnic populations in Clinical Trials Guidance for Industry; 2022. Available from: https://www.fda.gov/media/157635/download. Accessed September 20, 2023.
   Google Scholar
• Dreyer NA. Strengthening evidence-based medicine with real-world evidence. Lancet Healthy Longev. 2022;3(10):e641–e642. doi:10.1016/S2666-7568(22)00214-8
   PubMedGoogle Scholar
• Marc BMO, Daniel G, Frank K, et al. A framework for regulatory use of real-world evidence. White Paper. Duke Margolis Center for Health Policy; 2017. Available from: https://healthpolicy.duke.edu/publications/framework-regulatory-use-real-world-evidence. Accessed September 20, 2023.
   Google Scholar
• Curtis JP. Association of physician certification and outcomes among patients receiving an implantable cardioverter-defibrillator. JAMA. 2009;301(16):1661. doi:10.1001/jama.2009.547
   PubMed Web of Science ®Google Scholar
• Heidenreich PA, Hernandez AF, Yancy CW, Liang L, Peterson ED, Fonarow GC. Get with the guidelines program participation, process of care, and outcome for medicare patients hospitalized with heart failure. Circ Cardiovasc Qual Outcomes. 2012;5(1):37–43. doi:10.1161/CIRCOUTCOMES.110.959122
   PubMed Web of Science ®Google Scholar
• Concato J, Corrigan-Curay J. Real-world evidence — where are we now? N Engl J Med. 2022;386(18):1680–1682. doi:10.1056/NEJMp2200089
   PubMed Web of Science ®Google Scholar
• Purpura CA, Garry EM, Honig N, Case A, Rassen JA. The role of real‐world evidence in FDA‐approved new drug and biologics license applications. Clin Pharmacol Ther. 2022;111(1):135–144. doi:10.1002/cpt.2474
   PubMed Web of Science ®Google Scholar
• Bakker E, Plueschke K, Jonker CJ, Kurz X, Starokozhko V, Mol PGM. Contribution of real‐world evidence in European Medicines Agency’s regulatory decision making. Clin Pharmacol Ther. 2023;113(1):135–151. doi:10.1002/cpt.2766
   PubMed Web of Science ®Google Scholar
• Li M, Chen S, Lai Y, et al. Integrating real-world evidence in the regulatory decision-making process: a systematic analysis of experiences in the US, EU, and China using a logic model. Front Med. 2021;8:669509. doi:10.3389/fmed.2021.669509
   Web of Science ®Google Scholar
• Storm NE, Chang W, Lin T-C, et al. A novel case study of the use of real-world evidence to support the registration of an osteoporosis product in China. Ther Innov Regul Sci. 2022;56(1):137–144. doi:10.1007/s43441-021-00342-4
   PubMed Web of Science ®Google Scholar
• Tan YY, Papez V, Chang WH, Mueller SH, Denaxas S, Lai AG. Comparing clinical trial population representativeness to real-world populations: an external validity analysis encompassing 43 895 trials and 5 685 738 individuals across 989 unique drugs and 286 conditions in England. Lancet Healthy Longev. 2022;3(10):e674–e689. doi:10.1016/S2666-7568(22)00186-6
   PubMedGoogle Scholar
• Simon R, Blumenthal G, Rothenberg M, et al. The role of nonrandomized trials in the evaluation of oncology drugs. Clin Pharmacol Ther. 2015;97(5):502–507. doi:10.1002/cpt.86
   PubMed Web of Science ®Google Scholar
• Topp MS, Gökbuget N, Zugmaier G, et al. Long-term survival of patients with relapsed/refractory acute lymphoblastic leukemia treated with blinatumomab. Cancer. 2021;127(4):554–559. doi:10.1002/cncr.33298
   PubMed Web of Science ®Google Scholar
• Gökbuget N, Kelsh M, Chia V, et al. Blinatumomab vs historical standard therapy of adult relapsed/refractory acute lymphoblastic leukemia. Blood Cancer J. 2016;6(9):e473–e473. doi:10.1038/bcj.2016.84
   PubMedGoogle Scholar
• European Medicines Agency. Product information – Brineura 150 mg solution for infusion; 2017. Available from: https://www.ema.europa.eu/en/documents/product-information/brineura-epar-product-information_en.pdf. Accessed 20 September, 2023.
   Google Scholar
• American Heart Association. Get with the guidelines - coronary artery disease. Available from: https://www.heart.org/en/professional/quality-improvement/get-with-the-guidelines/get-with-the-guidelines-coronary-artery-disease. Accessed September 20, 2023.
   Google Scholar
• Shankar R, Poole L, Halmos T, et al. Using AI to support evidence & market access strategy development. Presentation presented at: ISPOR; May 7–10; 2023; Boston, MA, USA.
   Google Scholar
• Hillner BE, Siegel BA, Shields AF, et al. Relationship between cancer type and impact of PET and PET/CT on intended management: findings of the National Oncologic PET Registry. J Nucl Med. 2008;49(12):1928–1935. doi:10.2967/jnumed.108.056713
   PubMed Web of Science ®Google Scholar
• Park JJH, Siden E, Zoratti MJ, et al. Systematic review of basket trials, umbrella trials, and platform trials: a landscape analysis of master protocols. Trials. 2019;20(1):572. doi:10.1186/s13063-019-3664-1
   PubMed Web of Science ®Google Scholar
• Gliklich RE, Dreyer NA, Leavy MB. Registries for evaluating patient outcomes: a user’s guide. AHRQ Publication No. 13(14)-EHC111 ; 2014.
   Google Scholar
• Velentgas PDN, Nourjah P, Smith SR, Torchia MM. Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. Agency for Health Care Research and Quality; 2013:EHC099.
   Google Scholar
• Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology. 1990;1(1):43–46. doi:10.1097/00001648-199001000-00010
   PubMedGoogle Scholar
• Bager P, Hvas CL, Dahlerup JF. Drug‐specific hypophosphatemia and hypersensitivity reactions following different intravenous iron infusions. Br J Clin Pharmacol. 2017;83(5):1118–1125. doi:10.1111/bcp.13189
   PubMed Web of Science ®Google Scholar
• Barda N, Dagan N, Ben-Shlomo Y, et al. Safety of the BNT162b2 mRNA Covid-19 vaccine in a nationwide setting. N Engl J Med. 2021;385(12):1078–1090. doi:10.1056/NEJMoa2110475
   PubMed Web of Science ®Google Scholar
• Bourla A. Moonshot: Inside Pfizer’s Nine-Month Race to Make the Impossible Possible. New York, NY: Harper Business; 2022.
   Google Scholar
• Patient-generated health data. Available from: https://www.healthit.gov/topic/scientific-initiatives/pcor/patient-generated-health-data-pghd. Accessed September 20, 2023.
   Google Scholar
• Dreyer N, Reynolds MW, Albert L, et al. How frequent are acute reactions to COVID-19 vaccination and who is at risk? Vaccine. 2022;40(12):1904–1912. doi:10.1016/j.vaccine.2021.12.072
   PubMed Web of Science ®Google Scholar
• Reynolds MW, Xie Y, Knuth KB, et al. COVID-19 vaccination breakthrough infections in a real-world setting: using community reporters to evaluate vaccine effectiveness. Infect Drug Resist. 2022;15:5167–5182. doi:10.2147/IDR.S373183
   PubMed Web of Science ®Google Scholar
• Reynolds MW, Secora A, Joules A, et al. Evaluating real-world COVID-19 vaccine effectiveness using a test-negative case–control design. J Comp Eff Res. 2022;11(16):1161–1172. doi:10.2217/cer-2022-0069
   PubMed Web of Science ®Google Scholar
• Rothman KJ, Gallacher JE, Hatch EE. Why representativeness should be avoided. Int J Epidemiol. 2013;42(4):1012–1014. doi:10.1093/ije/dys223
   PubMed Web of Science ®Google Scholar
• Mack CD, Wasserman EB, Perrine CG, et al. Implementation and evolution of mitigation measures, testing, and contact tracing in the National Football League, August 9–November 21, 2020. MMWR Morb Mortal Wkly Rep. 2021;70(4):130–135. doi:10.15585/mmwr.mm7004e2
   PubMed Web of Science ®Google Scholar
• Mack CD, Osterholm M, Wasserman EB, et al. Optimizing SARS-CoV-2 surveillance in the United States: insights from The National Football League occupational health program. Ann Intern Med. 2021;174(8):1081–1089. doi:10.7326/M21-0319
   PubMed Web of Science ®Google Scholar
• Mack CD, Wasserman EB, Hostler CJ, et al. Effectiveness and use of reverse transcriptase polymerase chain reaction point of care testing in a large‐scale COVID ‐19 surveillance system. Pharmacoepidemiol Drug Saf. 2022;31(5):511–518. doi:10.1002/pds.5424
   PubMed Web of Science ®Google Scholar
• Mack CD, DiFiori J, Tai CG, et al. SARS-CoV-2 transmission risk among national basketball association players, staff, and vendors exposed to individuals with positive test results after COVID-19 recovery during the 2020 regular and postseason. JAMA Intern Med. 2021;181(7):960. doi:10.1001/jamainternmed.2021.2114
   PubMed Web of Science ®Google Scholar
• Tai CG, Maragakis LL, Connolly S, et al. Association between COVID-19 booster vaccination and omicron infection in a highly vaccinated cohort of players and staff in the National Basketball Association. JAMA. 2022;328(2):209. doi:10.1001/jama.2022.9479
   PubMed Web of Science ®Google Scholar
• Simon GE, Bindman AB, Dreyer NA, et al. When can we trust real‐world data to evaluate new medical treatments? Clin Pharmacol Ther. 2022;111(1):24–29. doi:10.1002/cpt.2252
   PubMed Web of Science ®Google Scholar
• Nicholas MNH, Silcox C, Aten A, et al. Determining real-world data’s fitness for use and the role of reliability. White Paper; 2019 https://healthpolicy.duke.edu/sites/default/files/2019-11/rwd_reliability.pdf.
   Google Scholar
• Dreyer NA, Mack CD, Anderson RB, Wojtys EM, Hershman EB, Sills A. Lessons on Data collection and curation from the NFL injury surveillance program. Sports Health. 2019;11(5):440–445. doi:10.1177/1941738119854759
   PubMedGoogle Scholar
• Kahn MG, Callahan TJ, Barnard J, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. eGEMs. 2016;4(1):18. doi:10.13063/2327-9214.1244
   Google Scholar
• Informatics OHDSa. Standardized data: the OMOP common data model. Available from: https://www.ohdsi.org/data-standardization/#:~:text=The%20Observational%20Medical%20Outcomes%20Partnership,that%20can%20produce%20reliable%20evidence. Accessed September 20, 2023.
   Google Scholar
• Osterman TJ, Terry M, Miller RS. Improving cancer data interoperability: the promise of the Minimal Common Oncology Data Elements (mCODE) initiative. JCO Clin Cancer Inform. 2020;4(4):993–1001. doi:10.1200/CCI.20.00059
   PubMedGoogle Scholar
• Daniel G, Silcox C, Bryan J, et al. Characterizing RWD quality and relevancy for regulatory purposes; 2018 https://healthpolicy.duke.edu/sites/default/files/2020-08/Characterizing%20RWD%20for%20Regulatory%20Use.pdf.
   Google Scholar
• US Food and Drug Administration. Demonstrating substantial evidence of effectiveness for human drug and biological products; 2019. Available from: https://www.fda.gov/media/133660/download. Accessed September 20, 2023.
   Google Scholar
• Dreyer NA, Sheth N, Trontell A, Gliklich RE. Good practices for handling adverse events detected through patient registries. Drug Inf J. 2008;42(5):421–428. doi:10.1177/009286150804200502
   Google Scholar
• Niu X, Divino V, Sharma S, Dekoven M, Anupindi VR, Dembek C. Healthcare resource utilization and exacerbations in patients with chronic obstructive pulmonary disease treated with nebulized glycopyrrolate in the USA: a real-world data analysis. J Med Econ. 2021;24(1):1–9. doi:10.1080/13696998.2020.1845185
   PubMed Web of Science ®Google Scholar
• Rivera DR, Gokhale MN, Reynolds MW, et al. Linking electronic health data in pharmacoepidemiology: appropriateness and feasibility. Pharmacoepidemiol Drug Saf. 2020;29(1):18–29. doi:10.1002/pds.4918
   PubMed Web of Science ®Google Scholar
• Pratt NL, Mack CD, Meyer AM, et al. Data linkage in pharmacoepidemiology: a call for rigorous evaluation and reporting. Pharmacoepidemiol Drug Saf. 2020;29(1):9–17. doi:10.1002/pds.4924
   PubMed Web of Science ®Google Scholar
• Mack CD, Meisel P, Herzog MM, et al. The establishment and refinement of the national basketball association player injury and illness database. J Athl Train. 2019;54(5):466–471. doi:10.4085/1062-6050-18-19
   PubMed Web of Science ®Google Scholar
• Mack C, Sendor RR, Solomon G, et al. Enhancing concussion management in the national football league: evolution and initial results of the unaffiliated neurotrauma consultants program, 2012–2017. Neurosurgery. 2020;87(2):312–319. doi:10.1093/neuros/nyz481
   PubMed Web of Science ®Google Scholar
• Mack C, Myers E, Barnes R, Solomon G, Sills A. Engaging athletic trainers in concussion detection: overview of the National Football League ATC spotter program, 2011–2017. J Athl Train. 2019;54(8):852–857. doi:10.4085/1062-6050-181-19
   PubMed Web of Science ®Google Scholar
• Hay JA, Kissler SM, Fauver JR, et al. Quantifying the impact of immune history and variant on SARS-CoV-2 viral kinetics and infection rebound: a retrospective cohort study. Elife. 2022;11. doi:10.7554/eLife.81849
   Web of Science ®Google Scholar
• Kissler SM, Fauver JR, Mack C, et al. Viral dynamics of acute SARS-CoV-2 infection and applications to diagnostic and public health strategies. PLoS Biol. 2021;19(7):e3001333. doi:10.1371/journal.pbio.3001333
   PubMed Web of Science ®Google Scholar
• Ferreira JC, Patino CM. Choosing wisely between randomized controlled trials and observational designs in studies about interventions. J Brasileiro de Pneumologia. 2016;42(3):165. doi:10.1590/S1806-37562016000000152
   PubMed Web of Science ®Google Scholar
• Lauer MS, D’Agostino RB. The randomized registry trial — the next disruptive technology in clinical research? N Engl J Med. 2013;369(17):1579–1581. doi:10.1056/NEJMp1310102
   PubMed Web of Science ®Google Scholar
• US Department of Veterans Affairs. VA cooperative studies program; 2018. Available from: https://www.vacsp.research.va.gov/CSP_597/Information_for_VA_Providers.asp#:~:text=The%20Diuretic%20Comparison%20Project%20%28DCP%29%20is%20the%20first,or%20more%20approved%20treatments%20when%20clinical%20equipoise%20exists. Accessed September 20, 2023.
   Google Scholar
• Alphs L, Benson C, Cheshire-Kinney K, et al. Real-world outcomes of paliperidone palmitate compared to daily oral antipsychotic therapy in schizophrenia: a randomized, open-label, review board–blinded 15-month study. J Clin Psychiatry. 2015;76(5):554–561. doi:10.4088/JCP.14m09584
   PubMed Web of Science ®Google Scholar
• Baumfeld Andre E, Reynolds R, Caubel P, Azoulay L, Dreyer NA. Trial designs using real‐world data: the changing landscape of the regulatory approval process. Pharmacoepidemiol Drug Saf. 2020;29(10):1201–1212. doi:10.1002/pds.4932
   PubMed Web of Science ®Google Scholar
• DeMonaco HJ, Ali A, Hippel EV. The major role of clinicians in the discovery of off-label drug therapies. Pharmacotherapy. 2006;26(3):323–332. doi:10.1592/phco.26.3.323
   PubMed Web of Science ®Google Scholar
• Gatto NM, Campbell UB, Rubinstein E, et al. The structured process to identify fit‐for‐purpose data: a data feasibility assessment framework. Clin Pharmacol Ther. 2022;111(1):122–134. doi:10.1002/cpt.2466
   PubMed Web of Science ®Google Scholar
• Hall GC, Sauer B, Bourke A, Brown JS, Reynolds MW, Casale RL. Guidelines for good database selection and use in pharmacoepidemiology research. Pharmacoepidemiol Drug Saf. 2012;21(1):1–10. doi:10.1002/pds.2229
   PubMed Web of Science ®Google Scholar
• Hernán MA, Wang W, Leaf DE. Target trial emulation: a framework for causal inference from observational data. JAMA. 2022;328(24):2446. doi:10.1001/jama.2022.21383
   PubMed Web of Science ®Google Scholar
• US Food and Drug Administration. Paclitaxel-coated balloons and stents for peripheral arterial disease. Available from: https://www.fda.gov/medical-devices/cardiovascular-devices/paclitaxel-coated-balloons-and-stents-peripheral-arterial-disease. Accessed September 20, 2023.
   Google Scholar
• Rothman KJ, Loughlin JE, Funch DP, Dreyer NA. Overall mortality of cellular telephone customers. Epidemiology. 1996;7(3):303–305. doi:10.1097/00001648-199605000-00015
   PubMed Web of Science ®Google Scholar
• Miksad RA, Abernethy AP. Harnessing the power of Real-World Evidence (RWE): a checklist to ensure regulatory-grade data quality. Clin Pharmacol Ther. 2018;103(2):202–205. doi:10.1002/cpt.946
   PubMed Web of Science ®Google Scholar
• Vandenbroucke JP. Strega, Strobe, Stard, Squire, Moose, Prisma, Gnosis, Trend, Orion, Coreq, Quorom, Remark… and Consort: for whom does the guideline toll? J Clin Epidemiol. 2009;62(6):594–596. doi:10.1016/j.jclinepi.2008.12.003
   PubMed Web of Science ®Google Scholar
• Roche N, Campbell JD, Krishnan JA, et al. Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group—European Academy of Allergy and Clinical Immunology Task Force. Clin Transl Allergy. 2019;9(1):20. doi:10.1186/s13601-019-0255-x
   PubMedGoogle Scholar
• Allen A, Patrick H, Ruof J, et al. Development and pilot test of the registry evaluation and quality standards tool: an information technology–based tool to support and review registries. Value Health. 2022;25(8):1390–1398. doi:10.1016/j.jval.2021.12.018
   PubMed Web of Science ®Google Scholar
• Dreyer NA, Bryant A, Velentgas P. The GRACE checklist: a validated assessment tool for high quality observational studies of comparative effectiveness. J Manag Care Special Pharm. 2016;22(10):1107–1113. doi:10.18553/jmcp.2016.22.10.1107
   PubMed Web of Science ®Google Scholar