Parallel and distributed clustering framework for big spatial data mining

ABSTRACT

Clustering techniques are very attractive for identifying and extracting patterns of interests from datasets. However, their application to very large spatial datasets presents numerous challenges such as high-dimensionality, heterogeneity, and high complexity of some algorithms. Distributed clustering techniques constitute a very good alternative to the Big Data challenges (e.g., Volume, Variety, Veracity, and Velocity). In this paper, we developed and implemented a Dynamic Parallel and Distributed clustering (DPDC) approach that can analyse Big Data within a reasonable response time and produce accurate results, by using existing and current computing and storage infrastructure, such as cloud computing. The DPDC approach consists of two phases. The first phase is fully parallel and it generates local clusters and the second phase aggregates the local results to obtain global clusters. The aggregation phase is designed in such a way that the final clusters are compact and accurate while the overall process is efficient in time and memory allocation. DPDC was thoroughly tested and compared to well-known clustering algorithms BIRCH and CURE. The results show that the approach not only produces high-quality results but also scales up very well by taking advantage of the Hadoop MapReduce paradigm or any distributed system.

GRAPHICAL ABSTRACT

KEYWORDS:

• Big Data
• MapReduce
• Hadoop
• spatial data mining
• clustering
• distributed clustering
• parallel clustering
• DBSCAN
• dynamic K-means

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The research work is conducted in the Insight Centre for Data Analytics, which is supported by Science Foundation Ireland [grant number SFI/12/RC/2289].