Please use this identifier to cite or link to this item:
|Distributed training for multi-layer neural networks by consensus
Engineering::Electrical and electronic engineering
|Liu, B., Ding, Z. & Lv, C. (2019). Distributed training for multi-layer neural networks by consensus. IEEE Transactions On Neural Networks and Learning Systems, 31(5), 1771-1778. https://dx.doi.org/10.1109/TNNLS.2019.2921926
|IEEE Transactions on Neural Networks and Learning Systems
|Over the past decade, there has been a growing interest in large-scale and privacy-concerned machine learning, especially in the situation where the data cannot be shared due to privacy protection or cannot be centralized due to computational limitations. Parallel computation has been proposed to circumvent these limitations, usually based on the master-slave and decentralized topologies, and the comparison study shows that a decentralized graph could avoid the possible communication jam on the central agent but incur extra communication cost. In this brief, a consensus algorithm is designed to allow all agents over the decentralized graph to converge to each other, and the distributed neural networks with enough consensus steps could have nearly the same performance as the centralized training model. Through the analysis of convergence, it is proved that all agents over an undirected graph could converge to the same optimal model even with only a single consensus step, and this can significantly reduce the communication cost. Simulation studies demonstrate that the proposed distributed training algorithm for multi-layer neural networks without data exchange could exhibit comparable or even better performance than the centralized training model.
|School of Mechanical and Aerospace Engineering
|© 2019 IEEE. All rights reserved.
|Appears in Collections:
|MAE Journal Articles
Updated on Mar 1, 2024
Web of ScienceTM
Updated on Oct 28, 2023
Updated on Mar 4, 2024
Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.