Programming models and systems for Big Data analysis

ABSTRACT

Big Data analysis refers to advanced and efficient data mining and machine learning techniques applied to large amount of data. Research work and results in the area of Big Data analysis are continuously rising, and more and more new and efficient architectures, programming models, systems, and data mining algorithms are proposed. Taking into account the most popular programming models for Big Data analysis (MapReduce, Directed Acyclic Graph, Message Passing, Bulk Synchronous Parallel, Workflow and SQL-like), we analysed the features of the main systems implementing them. Such systems are compared using four classification criteria (i.e. level of abstraction, type of parallelism, infrastructure scale and classes of applications) for helping developers and users to identify and select the best solution according to their skills, hardware availability, productivity and application needs.

GRAPHICAL ABSTRACT

This figure is a word cloud highlighting the most popular words related to Big Data analysis.

KEYWORDS:

• Parallel programming models
• programming systems
• Big Data analysis
• cloud computing
• programming frameworks
• MapReduce
• directed acyclic graph
• message passing
• bulk synchronous parallel
• workflow
• SQL-like

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. http://www.nist.gov/itl/bigdata/bigdatainfo.cfm

2. https://aws.amazon.com/elasticmapreduce/

3. https://azure.microsoft.com/services/hdinsight/

4. https://www.ibm.com/analytics/us/en/technology/hadoop/

5. https://wiki.openstack.org/wiki/Sahara

6. https://hadoop.apache.org/

7. https://spark.apache.org

8. http://mesos.apache.org/

9. https://storm.apache.org

10. https://flink.apache.org

11. https://azure.microsoft.com/en-us/services/machine-learning/

12. https://giraph.apache.org

13. https://netty.io

14. https://hama.apache.org

15. https://pig.apache.org

16. https://hive.apache.org