A Realistic Evaluation of Methods for Handling Missing Data When There is a Mixture of MCAR, MAR, and MNAR Mechanisms in the Same Dataset

Abstract

The impact of missing data on statistical inference varies depending on several factors such as the proportion of missingness, missing-data mechanism, and method employed to handle missing values. While these topics have been extensively studied, most recommendations have been made assuming that all missing values are from the same missing-data mechanism. In reality, it is very likely that a mixture of missing-data mechanisms is responsible for missing values in a dataset and even within the same pattern of missingness. Although a mixture of missing-data mechanisms and causes within a dataset is a likely scenario, the performance of popular missing-data methods under these circumstances is unknown. This study provides a realistic evaluation of methods for handling missing data in this setting using Monte Carlo simulation in the context of regression. This study also seeks to identify acceptable proportions of missing values that violate the missing-data mechanism assumed by the method used to handle missing values. Results indicate that multiple imputation (MI) performs better than other principled or ad-hoc methods. Different missing-data methods are also compared via the analysis of a real dataset in which mixtures of missingness mechanisms are created. Recommendations are provided for the use of different methods in practice.

KEYWORDS:

• Missing-data mechanisms
• nonignorable missingness
• multiple imputation
• maximum likelihood
• selection models

Article information

Conflict of interest disclosures: Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.

Ethical principles: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.

Funding: This work was supported by Grant R305D210023 from the Department of Education.

Role of the funders/sponsors: None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments: The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors’ institutions or the Department of Education is not intended and should not be inferred.

Notes

1 We thank our anonymous reviewers for suggesting such examples.

2 The “Equal Parts” condition in Table 1 refers to the condition in which there is an equal proportion of the three missing-data mechanisms.

3 This formula is given by equation (7.58) in Rencher and Schaalje (2008) which is denoted as $R_a^2.$

4 An exception occurs under the “50% MCAR” condition for an overall missingness rate of 40%, in which the bias is in an acceptable rather than an ideal range.

5 Exceptions occur for MI and NML under the “80% MCAR condition” with 40% missingness and for MI, NML, and RF under the “80% MAR” condition with 40% missingness.

6 In other words and parallel to equation 5, ${\psi }_y={\psi }_1={\psi }_2={\psi }_3$

7 There are two exceptions to this trend: 1) MI estimates of β₂ are positively biased under the “80% MCAR” condition, and 2) RF estimates of β₂ are in an acceptable range under the “80% MNAR” condition.