Please use this identifier to cite or link to this item:
https://hdl.handle.net/10356/65309
Full metadata record
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Xu, Yongliang | en |
dc.date.accessioned | 2015-07-16T03:45:23Z | en |
dc.date.available | 2015-07-16T03:45:23Z | en |
dc.date.copyright | 2015 | en |
dc.date.issued | 2015 | en |
dc.identifier.citation | Xu, Y. (2015). Hadoop job scheduling with dynamic task splitting. Master’s thesis, Nanyang Technological University, Singapore. | en |
dc.identifier.uri | https://hdl.handle.net/10356/65309 | en |
dc.description.abstract | Job scheduling affects the fairness and performance of shared Hadoop clusters. Fairness measures how fair the resources in the cluster are shared among different users in the Hadoop cluster. In Hadoop, schedulers will always attempt to maximize data locality. Data locality refers to the processing of data by tasks on nodes where the data is stored. Processing of data on data-local nodes improves performance, as there is no need to transfer data from one node to another. However, fairness and data locality are often in conflict. During scheduling, it is not always possible that the available nodes contain the data that a user’s job requires. In such cases, a scheduler may choose to schedule the tasks on these nodes regardless of data locality thus sacrificing performance. Alternatively, a scheduler may choose to give up the user’s slot and wait for a data-local node thus sacrificing fairness. Achieving pure fairness may compromise the data locality of the tasks that will in turn negatively affects performances, and vice-versa. Delay scheduling is a technique that attempts to improve data locality by waiting for a data-local node to be available. It violates the fairness criteria. The Dynamic Task Splitting Scheduler (DTSS) is proposed to mitigate the tradeoffs between fairness and data locality during job scheduling. DTSS does so by dynamically splitting a task and executing the split task immediately, on a non-data-local node, to improve the fairness. Analysis and experiments results show that it is possible to improve both fairness and the performance by adjusting the proportion of the task split. DTSS is shown to improve the makespan of different users in a cluster by 2% to 11% as compared to delay scheduling under conditions that is difficult to obtain data-local nodes on a cluster. Lastly, experiments show that DTSS is not a suitable scheduler under conditions where jobs are able to obtain data-local nodes easily. | en |
dc.format.extent | 68 p. | en |
dc.language.iso | en | en |
dc.subject | DRNTU::Engineering::Computer science and engineering::Information systems::Information systems applications | en |
dc.title | Hadoop job scheduling with dynamic task splitting | en |
dc.type | Thesis | en |
dc.contributor.supervisor | Cai Wentong | en |
dc.contributor.school | School of Computer Engineering | en |
dc.description.degree | MASTER OF ENGINEERING (SCE) | en |
dc.contributor.research | Parallel and Distributed Computing Centre | en |
dc.identifier.doi | 10.32657/10356/65309 | en |
item.fulltext | With Fulltext | - |
item.grantfulltext | open | - |
Appears in Collections: | SCSE Theses |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Hadoop Job Scheduling with Dynamic Task Splitting.pdf | 1.48 MB | Adobe PDF | View/Open |
Page view(s) 50
575
Updated on Mar 27, 2024
Download(s) 10
423
Updated on Mar 27, 2024
Google ScholarTM
Check
Altmetric
Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.