Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/105436
Title: Time- and cost- efficient task scheduling across geo-distributed data centers
Authors: Hu, Zhiming
Li, Baochun
Luo, Jun
Keywords: Big Data Processing
Task Scheduling
DRNTU::Engineering::Computer science and engineering
Issue Date: 2017
Source: Hu, Z., Li, B., & Luo, J. (2018). Time- and cost- efficient task scheduling across geo-distributed data centers. IEEE Transactions on Parallel and Distributed Systems, 29(3), 705-718. doi:10.1109/TPDS.2017.2773504
Series/Report no.: IEEE Transactions on Parallel and Distributed Systems
Abstract: Typically called big data processing, analyzing large volumes of data from geographically distributed regions with machine learning algorithms has emerged as an important analytical tool for governments and multinational corporations. The traditional wisdom calls for the collection of all the data across the world to a central data center location, to be processed using data-parallel applications. This is neither efficient nor practical as the volume of data grows exponentially. Rather than transferring data, we believe that computation tasks should be scheduled near the data, while data should be processed with a minimum amount of transfers across data centers. In this paper, we design and implement Flutter, a new task scheduling algorithm that reduces both the completion times and the network costs of big data processing jobs across geographically distributed data centers. To cater to the specific characteristics of data-parallel applications, in the case of optimizing the job completion times only, we first formulate our problem as a lexicographical min-max integer linear programming (ILP) problem, and then transform the ILP problem into a nonlinear program problem with a separable convex objective function and a totally unimodular constraint matrix, which can be further solved using a standard linear programming solver efficiently in an online fashion. In the case of improving both time-and costefficiency, we formulate the general problem as an ILP problem and we find out that solving an LP problem can achieve the same goal in the real practice. Our implementation of Flutter is based on Apache Spark, a modern framework popular for big data processing. Our experimental results have shown convincing evidence that Flutter can shorten both job completion times and network costs by a substantial margin.
URI: https://hdl.handle.net/10356/105436
http://hdl.handle.net/10220/48661
ISSN: 1045-9219
DOI: 10.1109/TPDS.2017.2773504
Rights: © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/TPDS.2017.2773504
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Journal Articles

Files in This Item:
File Description SizeFormat 
Time- and Cost- Efficient Task Scheduling.pdf486.44 kBAdobe PDFThumbnail
View/Open

SCOPUSTM   
Citations 10

24
Updated on Mar 3, 2021

PublonsTM
Citations 20

11
Updated on Mar 7, 2021

Page view(s)

50
Updated on Jun 15, 2021

Download(s) 50

65
Updated on Jun 15, 2021

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.