https://orcid.org/0000-0001-7515-6526
Abstract
Randomized, controlled, double-blinded studies, in which treated subjects are randomly selected from the same pool as controlled (untreated) ones and neither the caregiver nor the patient knows which is which, are widely accepted as the gold standard of experimental medicine. There are well-documented advantages of such studies. There are, however, significant limitations of them as well of which it is important to be aware. Notably, physicians who rely on experience and on what they were taught in medical school and post-graduate training are not necessarily wrong when this information runs contrary to the results of such studies. Some limitations of them are widely known and taught, such as inadequate sample size, failure of proper randomization, et cetera; others are less well-known. We shall focus on the latter.
Limitations of studies that produce positive results
A critical, but often unstated or unappreciated assumption in clinical experience as well as of randomized as well as not so randomized studies is that the two (or more) groups compared or studied, encompassing normal/control and treated groups, are relatively uniform within themselves as well as between groups. If this is incorrect this can, of course, generate false-positive results. An outstanding example, familiar to many physicians, was the idea that bullous pemphigoid is a cutaneous sign of internal malignancy. It turned out that this information was based primarily on observations of populations in everyday medical practice without attention being paid to the ages of the patients. Bullous pemphigoid is more common in the elderly, as is internal malignancyCitation1. When older populations with and without bullous pemphigoid were studied the association of age with bullous pemphigoid disappeared. Similarly, it was once thought that those that entered the specialty of radiation therapy were at increased risk to develop myelogenous leukemiaCitation2; later studies showed that this was not increased using age-matched controls. There is a well-documented spike in the incidence of this disease in young adults, the age of most physicians in this new field at the timeCitation3. Notably, the correct answer in both instances emerged from controlling for the age of the patients; however, this required that the confounding variable, in this case, age, is controlled for. Note that if a different variable, such as sexual activity, possibly not controlled for, is relevant, the study may still provide an unreliable answer depending on what specific question is asked. It is well to recall the well-known advice of Dr. Philip Leder (Director, Laboratory of Molecular Genetics, NICHD, NIH) that “The most dangerous factor is the one no one is thinking about.”
Limitations of studies that produce negative results
A more challenging problem arises in analyzing data that fail to show a difference between the two groups. Here heterogeneity within groups can be a major issue.
The statistical model of controlled studies assumes homogeneity within each group, usually the control group and the test group. When this is not true, it may be much harder to find a statistical difference even if it is not only present but marked. A glaring example is studies of the effectiveness of various treatments in treating COVID-19. A number of the studies have failed to show a statistically significant benefit, in contrast to anecdotal reports and some physicians’ experience: are these studies reliable? As one of the authors (WCL) has previously noted, COVID-19 is a complex disease, with a succession of stages that may be of varying severity in different patients. Thus, if the agent is applied at a different stage in the course of the disease than that (or those) at which it is effective, it might not be expected to be found to be of benefit even if it were of major benefit if applied at a different stageCitation4.
Randomized, double-blinded studies have their value, but they are not perfect. It is well to remember the adage “Failure to prove a difference does not prove lack of a difference.”
Limitations of studies that fortuitously produce positive results
Several studies have shown positive results not for treating the disease under study but fortuitously for treating or preventing another disorder that was arising or was present within the same population. Two examples come immediately to mind: In one, investigators testing a new drug for restoration of hair loss were told by men in their study population that regardless of whether they grew hair their erectile function was markedly improved. This led to further studies targeted to this issue that resulted in the drug sidenophil. On the other, several observations have been reported that certain drugs, notably anti-parasitic and antimalarial medications, may be useful against COVID-19Citation5. In at least one study a population of nursing home residents was treated with a drug in a randomized, double-blinded study to test whether it was effective against scabiesCitation6. During the study, several patients in the control group developed COVID-19 whereas none in the treated group did. Neither study, by itself, is reliable, because the effect observed was not the one under study. Suppose the patients in the control populations of either study had developed zoster, pemphigoid, or any of a hundred other diseases that the study populations had not developed, would those studies, by themselves, have reliably indicated that the agent under study was an effective treatment for any of these other disorders? This is like asking whether two individuals in a crowded room have the same birthday. As long as we do not specify which birthday, the odds are pretty high, whereas if we ask if any of them have your birthday, the odds are very much lower.
Conclusion
No study design is perfect, just as no conclusion based on clinical experience is infallible. In the experience of one of us (WCL), some physicians rely exclusively on results of randomized, blinded, controlled studies for practice guidelines, whereas others disregard them altogether and rely only on their clinical experience and/or on what they were taught during their education/training. We suggest that neither approach may be wise and that the limitations of both approaches be kept in mind.
Transparency
Declaration of funding
This paper was not funded.
Declaration of financial/other relationships
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.
Acknowledgements
None.
References
- Ortíz LJ, Vázquez M, Sánchez JL. Bullous pemphigoid and malignancy. Bol Asoc Med P R. 1990;82(10):458–459.
- March HC. Leukemia in radiologists. Radiology. 1944;43(3):275–278.
- Ribera J-M, Oriol A. Acute lymphoblastic leukemia in adolescents and young adults. Hematol Oncol Clin. 2009;23(5):1033–1042.
- Lambert WC, Alhatem A, Lambert MW, et al. What we have learned from the COVID-19 pandemic: time to think outside the box, maybe far outside. Clin Dermatol. 2021;39(1):107–117.
- Sivabakya TK, Srinivas G. Will the antimalarial drug take over to combat COVID-19? J Public Health. 2020;30:241–244.
- Bernigaud C, Guillemot D, Ahmed-Belkacem A, et al. Oral ivermectin for a scabies outbreak in a long-term care facility: potential value in preventing COVID-19 and associated mortality. Br J Dermatol. 2021;184(6):1207–1209.