Statistical analysis of multi job processing in Hadoop environment using schedulers

Abstract

Hadoop schedulers facilitate effective sharing of system resources among multiple tasks for the purpose of effective completion of jobs. Upon submission of a job, the same will be transferred onto a queue by Hadoop scheduler. Once the transfer is complete, aforesaid job will be split up into multiple tasks and is further distributed among various computing nodes. Herein, performance is directly related to time taken up by aforesaid tasks to complete the assigned job. Optimum distribution of system resources among such tasks play a vital role in reduction of job completion time, and thus can ensure improved performance in relation to the same. Hadoop, by virtue uses FIFO scheduler. In the proposed experiment, FIFO scheduler’s job execution time is discussed and compared with the job execution time of Fair scheduler and Capacity scheduler.

Subject Classification:

• 6804
• 9006

Keywords:

• Big data
• Hadoop
• MapReduce
• FIFO scheduler
• FAIR scheduler and capacity scheduler