Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/65309
Title: Hadoop job scheduling with dynamic task splitting
Authors: Xu, Yongliang
Keywords: DRNTU::Engineering::Computer science and engineering::Information systems::Information systems applications
Issue Date: 2015
Source: Xu, Y. (2015). Hadoop job scheduling with dynamic task splitting. Master’s thesis, Nanyang Technological University, Singapore.
Abstract: Job scheduling affects the fairness and performance of shared Hadoop clusters. Fairness measures how fair the resources in the cluster are shared among different users in the Hadoop cluster. In Hadoop, schedulers will always attempt to maximize data locality. Data locality refers to the processing of data by tasks on nodes where the data is stored. Processing of data on data-local nodes improves performance, as there is no need to transfer data from one node to another. However, fairness and data locality are often in conflict. During scheduling, it is not always possible that the available nodes contain the data that a user’s job requires. In such cases, a scheduler may choose to schedule the tasks on these nodes regardless of data locality thus sacrificing performance. Alternatively, a scheduler may choose to give up the user’s slot and wait for a data-local node thus sacrificing fairness. Achieving pure fairness may compromise the data locality of the tasks that will in turn negatively affects performances, and vice-versa. Delay scheduling is a technique that attempts to improve data locality by waiting for a data-local node to be available. It violates the fairness criteria. The Dynamic Task Splitting Scheduler (DTSS) is proposed to mitigate the tradeoffs between fairness and data locality during job scheduling. DTSS does so by dynamically splitting a task and executing the split task immediately, on a non-data-local node, to improve the fairness. Analysis and experiments results show that it is possible to improve both fairness and the performance by adjusting the proportion of the task split. DTSS is shown to improve the makespan of different users in a cluster by 2% to 11% as compared to delay scheduling under conditions that is difficult to obtain data-local nodes on a cluster. Lastly, experiments show that DTSS is not a suitable scheduler under conditions where jobs are able to obtain data-local nodes easily.
URI: https://hdl.handle.net/10356/65309
DOI: 10.32657/10356/65309
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Theses

Files in This Item:
File Description SizeFormat 
Hadoop Job Scheduling with Dynamic Task Splitting.pdf1.48 MBAdobe PDFThumbnail
View/Open

Page view(s) 20

371
checked on Oct 29, 2020

Download(s) 20

158
checked on Oct 29, 2020

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.