Please use this identifier to cite or link to this item:
https://hdl.handle.net/10356/177576
Title: | Towards advanced distributed data processing: framework, optimization, and application | Authors: | Liu, Kaiqi | Keywords: | Computer and Information Science | Issue Date: | 2024 | Publisher: | Nanyang Technological University | Source: | Liu, K. (2024). Towards advanced distributed data processing: framework, optimization, and application. Doctoral thesis, Nanyang Technological University, Singapore. https://hdl.handle.net/10356/177576 | Abstract: | The surge in available big data has drawn significant interest in distributed processing methods capable of handling the ever-expanding data volume and increasing computational complexities efficiently and at scale. While existing distributed data processing frameworks, such as Apache Spark, have proven effective in various applications, there is still considerable room for improvement and exploration in this field. This thesis focuses on three key aspects of advancing distributed data processing using Apache Spark. First, a novel framework is introduced to extend Spark’s capabilities, enabling the efficient processing of large-scale spatio-temporal data to better serve machine-learning applications. This framework not only achieves high efficiency but also provides a user-friendly interface. Second, a deep-learning-based optimization approach tailored to enhance the efficiency of Spark SQL execution is proposed. The end-to-end system integration of this approach leads to practical performance gains. Last, a distributed solution for the computational-intensive large-scale microscopic crowd simulation is designed and implemented aiming to improve the scalability and efficiency of such applications. These three works collectively expand the application of distributed computing and enhance efficiency through the implementation of state-of-the-art techniques. | URI: | https://hdl.handle.net/10356/177576 | DOI: | 10.32657/10356/177576 | Schools: | School of Computer Science and Engineering | Research Centres: | Alibaba-NTU Singapore Joint Research Institute | Rights: | This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). | Fulltext Permission: | open | Fulltext Availability: | With Fulltext |
Appears in Collections: | SCSE Student Reports (FYP/IA/PA/PI) |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Liu_Kaiqi_thesis.pdf | 3.57 MB | Adobe PDF | ![]() View/Open |
Page view(s)
242
Updated on Feb 8, 2025
Download(s) 50
193
Updated on Feb 8, 2025
Google ScholarTM
Check
Altmetric
Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.