Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/80355
Title: Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing
Authors: Niu, Zhaojie
Tang, Shanjiang
He, Bingsheng
Keywords: Optimization
Processor scheduling
Adaptation models
Computational modeling
Issue Date: 2015
Source: Niu, Z., Tang, S., & He, B. (2015). Gemini: An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing. 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom), 66-73.
Conference: 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom)
Abstract: In data-intensive cluster computing platforms such as Hadoop YARN, performance and fairness are two important factors for system design and optimizations. Many previous studies are either for performance or for fairness solely, without considering the tradeoff between performance and fairness. Recent studies observe that there is a tradeoff between performance and fairness because of resource contention between users/jobs. However, their scheduling algorithms for bi-criteria optimization between performance and fairness are static, without considering the impact of different workload characteristics on the tradeoff between performance and fairness. In this paper, we propose an adaptive scheduler called Gemini for Hadoop YARN. We first develop a model with the regression approach to estimate the performance improvement and the fairness loss under the sharing computation compared to the exclusive non-sharing scenario. Next, we leverage the model to guide the resource allocation for pending tasks to optimize the performance of the cluster given the user-defined fairness level. Instead of using a static scheduling policy, Gemini adaptively decides the proper scheduling policy according to the current running workload. We implement Gemini in Hadoop YARN. Experimental results show that Gemini outperforms the state-of-the-art approach in two aspects. 1) For the same fairness loss, Gemini improves the performance by up to 225% and 200% in real deployment and the large-scale simulation, respectively, 2) For the same performance improvement, Gemini reduces the fairness loss up to 70% and 62.5% in real deployment and the large-scale simulation, respectively.
URI: https://hdl.handle.net/10356/80355
http://hdl.handle.net/10220/40532
DOI: 10.1109/CloudCom.2015.52
Schools: School of Computer Engineering 
Rights: © 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: [http://dx.doi.org/10.1109/CloudCom.2015.52].
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Conference Papers

Files in This Item:
File Description SizeFormat 
Gemini_An Adaptive Performance-Fairness Scheduler for Data-Intensive Cluster Computing.pdf890.47 kBAdobe PDFThumbnail
View/Open

SCOPUSTM   
Citations 20

13
Updated on Mar 27, 2024

Web of ScienceTM
Citations 20

11
Updated on Oct 25, 2023

Page view(s) 50

442
Updated on Mar 27, 2024

Download(s) 20

233
Updated on Mar 27, 2024

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.