Meta-Analyses Proved Inconsistent in How Missing Data Were Handled Across Their Included Primary Trials: A Methodological Survey

Figures & data

Table 1 Commonly Used Methods of Handling Missing Outcome Data Among Trial Participants

Method of Handling Missing Data	Implications for Participants with Missing Data in the Numerator and Denominator
	Intervention Arm		Control Arm
	Numerator	Denominator	Numerator	Denominator
Complete case analysis	Excluded	Excluded	Excluded	Excluded
Best case scenario	Assumed that all had a favorable outcome	Included	Assumed that all had an unfavorable outcome	Included
None of the participants with missing data had the outcome	Assumed that none had the outcome	Included	Assumed that none had the outcome	Included
All participants with missing data had the outcome	Assumed that all had the outcome	Included	Assumed that all had the outcome	Included
Worst case scenario	Assumed that all had an unfavorable outcome	Included	Assumed that all had a favorable outcome	Included

Table 2 Examples Illustrating How Meta-Analyses Addressing the Same Study Question Might Handle Missing Outcome Data for an Unfavorable Dichotomous Outcome from an RCT Report and Thus Informed Classification of the “Analytical Method Reviews Actually Used”

Intervention Arm						Control Arm
RCT report data
	Number of participants randomized	Number of events		Number of participants with missing data		Number of participants randomized	Number of events	Number of participants with missing data
RCT 1	100	5		2		100	10	5
Meta-analysis data
	Denominator		Numerator			Denominator	Numerator		Our classification of the actual analytical method by the SR
MA 1	98		5			95	10		Complete case analysis
MA 2	100		5			100	10		Assumed none had the event
MA 3	100		7			100	15		Assumed that all had the event
MA 4	100		7			100	10		Worst-case scenario
MA 5	100		5			100	15		Best-case scenario

Abbreviations: MA, meta-analysis; RCT, randomized controlled trial; SR, systematic review.

Table 3 Hypothetical Scenarios Illustrating the Process for Judging Consistency Between “Reported” and “Actual” Analytical Methods for Addressing Missing Outcome Data

		Reported Analytical Method	Analytical Method Actually Used	Analytical Method Actually Used Consistent Within the Meta-Analysis	Analytical Method Actually Used Consistent with Reported Analytical Method
Systematic review 1		Assume all had the event
RCT 1		–	Complete case analysis	No	Not applicable since the analytical method actually used were inconsistent across trials
RCT 2		–	Assume none had the event
RCT 3		–	Different methods for different categories of participants with missing data
Systematic review 2		Complete case analysis
RCT 4		–	Complete case analysis	Yes	Yes
RCT 5		–	Complete case analysis
RCT 6		–	Complete case analysis
Systematic review 3		Assume all had the event
RCT 7		–	Complete case analysis	Yes	No
RCT 8		–	Complete case analysis
RCT 9		–	Complete case analysis
Systematic review 4		Not reported
RCT 10		–	Complete case analysis	Yes	Not applicable since the reported analytical method is not available
RCT 11		–	Complete case analysis
RCT 12		–	Complete case analysis
Systematic review 5		Not reported
RCT 13		–	Complete case analysis	No	Not applicable since the reported analytical method is not available
RCT 14		–	Assume none had the event
RCT 15		–	Different methods for different categories of participants with missing data

Table 4 “Analytical Method Reviews Actually Used” to Handle Missing Data Across RCTs Included in Meta-Analysis

Variable	n (%)
Ability to classify the analytical method reviews actually used (n=638)
Able to classify	241 (37.8)
Not applicable (no missing data)	207 (32.4)
Could not be explained (numbers do not add up)	161 (25.2)
Wrong data extraction	5 (0.8)
No data available from RCT or SR	24 (3.8)
Classification of the analytical method reviews actually used (n= 241^+)
Complete case analysis	128 (53.1)
None of the participants with missing data had the event of interest	58 (24.1)
All the participants with missing data had the event of interest	2 (0.8)
Worst-case scenario	1 (0.4)
Best-case scenario	0
Same event rate as those followed up	0
Other	1 (0.4)
Different methods for different categories of participants with missing data	51 (21.2)
The SR authors used in the meta-analysis a denominator used by the RCT (n=431*)
Definitely yes	348 (80.7)
Definitely no	65 (15.1)
Unclear	18 (4.2)
The SR authors used in the meta-analysis a numerator used by the RCT (n=431*)
Definitely yes	345 (80.0)
Definitely no	53 (12.3)
Unclear	33 (7.7)

Notes: ⁺n=241 RCTs for which we could classify an “analytical method reviews actually used”. *n= 431 RCTs excluding those with no missing data (638–207 RCTs that had no missing data).

Abbreviations: ITT, intention-to-treat; LTFU, lost to follow-up; RCT, randomized controlled trials; SR, systematic reviews.

Figure 1 Consistency in analytical methods within the same meta-analysis and versus the reported analytical method.

Abbreviations: RCTs: randomized controlled trials; SR: systematic review.