Design and optimization of data center performance

Abstract

Distributed parallel file systems is a storage architecture used in many High Performance Computing (HPC) clusters. With the increasing need to run I/O and data intensive parallel applications in these HPC clusters, there will be a demand for higher processor performance and higher I/O bandwidth from the Distributed file system.

The development of RAM disk and Solid-State Disks (SSD) has become a new option to having a higher I/O bandwidth, but the cost of solely using such storage devices for a system is too high to be considered. Therefore, to harness the capabilities of these new drives, the hard disk can be used together with the RAM disk or SSD. This combination of different type of devices is known as the hybrid storage, it aims to improve the I/O performance but at a reasonable cost.

In this project, the author had to develop a pseudo file systems that utilizes the features of hybrid file system. This files system would serve as the backend of the file system that handles the I/O operations between different hard disk in the hybrid file system. The aim would be to better understand how such a system would work and how the program that would be used to build it works.

The second task of the project would be to create additional functions to an existing pseudo file system programmed in C. The existing file system has only read/write functions and a simple data migration method. The author will be implementing functions to monitor the usage of the file system and create functions to facilitate data migration between different devices.