Observational studies of treatment effectiveness: worthwhile or worthless?

Figures & data

Table 1 Important considerations in the design of cohort studies to evaluate treatment effectiveness and how to mitigate the risk of bias

Consideration	Bias	Definition	Problem	Approach that can help
Starting follow-up	Prevalent user bias	Bias that arises from the inclusion of individuals who have already initiated treatment prior to inclusion in study	Prevalent users survived a period of exposure before the study and may be at lower risk of an event	New-user design
Starting follow-up	Incomplete follow- up bias	Bias that arises owing to too short a period of follow-up for an outcome	If an event is due to a cumulative effect of a treatment, insufficient follow-up may not allow enough time for the event to occur	Prevalent user design
Comparator choice	Channeling bias	Bias that arises when the clinical indication for choosing a particular treatment also affects the outcome	If there is channeling bias then an effectiveness estimate may be biased, with a directionality based on whether the comparison group has better or worse prognosis	Use of an active comparator where there is equipoise in treatment choice
Identification and measurement of confounders	Confounding bias	Bias that arises when factors that affect both treatment choice and outcome are not accounted for	The directionality of this bias can be unpredictable and will depend on both accurate identification and measurement of confounders, as well as the extent of unmeasured confounding	Identification, accurate measurement and controlling for confounding
Outcome ascertainment	Reporting bias	Bias that arises owing to an imbalance in recording of an outcome across treatment groups	This can lead to a treatment appearing more or less effective when, in fact, the outcomes are simply recorded differently across treatment groups	Careful consideration and understanding of how an outcome might be detected and recorded in clinical practice
Outcome ascertainment	Attrition bias	Bias that arises owing to an imbalance in duration of follow-up across treatment groups	This can lead to lower numbers for an outcome in the group with shorter follow-up time	Ensuring comparison groups have similar follow-up time across which an outcome can realistically be recorded

Table 2 Comparison of baseline characteristics from three randomized controlled trials and the present case study

Study	Participants, n		Age (years), mean (SD)		Male, n (%)		HbA1c % (SD) [mmol/mol, SD]		Weight (kg), mean (SD)
	Sita	Sulf	Sita	Sulf	Sita	Sulf	Sita	Sulf	Sita	Sulf
Ahrén et al (2014)^Citation30	302	307	54.3 (9.8)	54.4 (10.0)	139 (46.0)	158 (51.5)	8.1 (0.8) [65, 8.7]	8.1 (0.8) [65, 8.7]	90.3 (19.1)	91.8 (20.4)
Arechavaleta et al (2011)^Citation31	516	519	56.3 (9.7)	56.2 (10.1)	284 (55.0)	279 (53.8)	7.5 (0.7) [58, 7.7]	7.5 (0.8) [58, 8.7]	80.6 (15.2)	82.0 (16.7)
Seck et al (2010)^Citation32	588	584	56.8 (9.3)	56.6 (9.8)	336 (57.1)	358 (61.3)	7.7 (0.9) [61, 9.8]	7.6 (0.9) [60, 9.8]	89.5 (17.4)	89.7 (17.5)
Sharma et al (present study)	3,306	15,880	58.9 (11.2)	61.4 (11.7)	1976 (59.8)	9695 (61.1)	8.6 (1.4) [70, 14.8]	9.0 (1.7) [74, 18.9]	99.6 (21.9)	91.4 (19.7)

Abbreviations: Sita, sitagliptin; Sulf, sulfonylureas.

Figure 1 Forest plot comparing our case study (Sharma et al) with meta-analyses of previous RCT examining change in HbA_1c (mmol/mol) between sitagliptin and sulfonylurea as add-on to metformin.

Source: Adapted from Sharma M, Beckley N, Nazareth I, Petersen I. Effectiveness of sitagliptin compared to sulfonylureas for type 2 diabetes mellitus inadequately controlled on metformin: a systematic review and meta-analysis. BMJ Open. 2017;7(10):e017260.²⁸

Notes: Weights, where present, are from fixed-effects meta-analysis (Mantel–Haenszel method), although random-effects estimates (DerSimonian–Laird method) were identical.
Abbreviations: Dur, duration; Mean diff, mean difference; Sita, sitagliptin; Sulf, sulfonylureas; Tot, total participants; RCT, randomized controlled trial; Obs, observational study; NA, not applicable.

Table 3 Results from the case study: analysis of mean difference in HbA_1c (mmol/mol) 12 months after initiation of sitagliptin vs sulfonylureas

Sitagliptin vs sulfonylurea	Unadjusted	Adjusted for baseline HbA1c	Adjusted for sex, age and baseline HbA1c	Fully adjusted multivariable model a
Population size (n=19,156)	0.55 (−0.04 to 1.13)	1.78 (1.23 to 2.33)	1.13 (0.59 to 1.67)	0.89 (0.33 to 1.45)

Notes: Data are shown as the mean difference (95% CI).

a Adjusted for potential confounders including baseline HbA_1c, baseline weight, age, year of cohort entry, face-to-face consultation frequency, year of entry, sex, Townsend deprivation score, smoking status, metformin dose, alcohol consumption, history of hypoglycemia, chronic kidney disease, neuropathy, heart failure, anemias, dementia, liver disease, arrhythmias, cancer, hypothyroidism, hyperthyroidism, pancreatitis, and having a prescription within 3 months of treatment initiation for antihypertensives, antiplatelets, anticoagulants, antiarrhythmics, diuretics, statins, other lipid-lowering drugs, antidepressants, antipsychotics, antiobesity drugs, oral or intravenous steroid medication, thyroxine, antithyroid drugs or anxiolytics. Individuals prescribed sulfonylureas are the reference population in all regression estimates.

AhrénBJohnsonSLStewartMHARMONY 3 Study GroupHARMONY 3: 104-week randomized, double-blind, placebo- and active-controlled trial assessing the efficacy and safety of albiglutide compared with placebo, sitagliptin, and glimepiride in patients with type 2 diabetes taking metforminDiabetes Care20143782141214824898304

 PubMed Web of Science ®Google Scholar

ArechavaletaRSeckTChenYEfficacy and safety of treatment with sitagliptin or glimepiride in patients with type 2 diabetes inadequately controlled on metformin monotherapy: a randomized, double-blind, non-inferiority trialDiabetes Obes Metab201113216016821199268

 PubMed Web of Science ®Google Scholar

SeckTNauckMShengDSitagliptin Study 024 GroupSafety and efficacy of treatment with sitagliptin or glipizide in patients with type 2 diabetes inadequately controlled on metformin: a 2-year studyInt J Clin Pract201064556257620456211

 PubMed Web of Science ®Google Scholar

SharmaMBeckleyNNazarethIPetersenIEffectiveness of sitagliptin compared to sulfonylureas for type 2 diabetes mellitus inadequately controlled on metformin: a systematic review and meta-analysisBMJ Open2017710e017260

 Web of Science ®Google Scholar