Composite quantile regression for massive datasets

ABSTRACT

Analysis of massive datasets is challenging owing to limitations of computer primary memory. Composite quantile regression (CQR) is a robust and efficient estimation method. In this paper, we extend CQR to massive datasets and propose a divide-and-conquer CQR method. The basic idea is to split the entire dataset into several blocks, applying the CQR method for data in each block, and finally combining these regression results via weighted average. The proposed approach significantly reduces the required amount of primary memory, and the resulting estimate will be as efficient as if the entire data set is analysed simultaneously. Moreover, to improve the efficiency of CQR, we propose a weighted CQR estimation approach. To achieve sparsity with high-dimensional covariates, we develop a variable selection procedure to select significant parametric components and prove the method possessing the oracle property. Both simulations and data analysis are conducted to illustrate the finite sample performance of the proposed methods.

Keywords:

• Massive dataset
• divide and conquer
• composite quantile regression
• variable selection

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This research was supported by the Fundamental Research Funds for the Central Universities (Series number: 2232016D3-17). Additional partial support is also provided by the Shanghai Sailing Program (Series number: 17YF1400800); the National Natural Science Foundation of China (Series number: 11626057); and the China Scholarship Council (Series number: 201706635019).