Box–Cox Transformation in Big Data

ABSTRACT

The Box–Cox transformation is an important technique in linear regression when assumptions of a regression model are seriously violated. The technique has been widely accepted and extensively applied since it was first proposed. Based on the maximum likelihood approach, previous methods and algorithms for the Box–Cox transformation are mostly developed for small or moderate data. These methods and algorithms cannot be applied to big data because of the memory and storage capacity barriers. To overcome these difficulties, the present article proposes new methods and algorithms, where the basic idea is to construct and compute a set of summary statistics, which is termed as the Box–Cox information array in the article. According to the property of the maximum likelihood approach, the computation of the Box–Cox information array is the only issue to be considered in reading of data. Once the Box–Cox information array is obtained, the optimal power transformation as well as the corresponding estimates of model parameters can be quickly computed. Since the whole dataset is scanned only once, the proposed methods and algorithms can be extremely efficient and fast even when multiple models are considered. It is expected that the basic knowledge gained in this article will have a great impact on the development of statistical methods and algorithms for big data.

KEYWORDS:

• Big data
• Box–Cox information array
• Linear regression
• MapReduce
• Maximum likelihood estimation
• Parallel computation

Acknowledgments

The authors appreciate suggestions and comments from the editor, associate editor, and three anonymous referees, which significantly improved the quality of the article.