Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/161253
Title: Distributed training for multi-layer neural networks by consensus
Authors: Liu, Bo
Ding, Zhengtao
Lv, Chen
Keywords: Engineering::Mechanical engineering
Engineering::Electrical and electronic engineering
Issue Date: 2019
Source: Liu, B., Ding, Z. & Lv, C. (2019). Distributed training for multi-layer neural networks by consensus. IEEE Transactions On Neural Networks and Learning Systems, 31(5), 1771-1778. https://dx.doi.org/10.1109/TNNLS.2019.2921926
Journal: IEEE Transactions on Neural Networks and Learning Systems 
Abstract: Over the past decade, there has been a growing interest in large-scale and privacy-concerned machine learning, especially in the situation where the data cannot be shared due to privacy protection or cannot be centralized due to computational limitations. Parallel computation has been proposed to circumvent these limitations, usually based on the master-slave and decentralized topologies, and the comparison study shows that a decentralized graph could avoid the possible communication jam on the central agent but incur extra communication cost. In this brief, a consensus algorithm is designed to allow all agents over the decentralized graph to converge to each other, and the distributed neural networks with enough consensus steps could have nearly the same performance as the centralized training model. Through the analysis of convergence, it is proved that all agents over an undirected graph could converge to the same optimal model even with only a single consensus step, and this can significantly reduce the communication cost. Simulation studies demonstrate that the proposed distributed training algorithm for multi-layer neural networks without data exchange could exhibit comparable or even better performance than the centralized training model.
URI: https://hdl.handle.net/10356/161253
ISSN: 2162-237X
DOI: 10.1109/TNNLS.2019.2921926
Schools: School of Mechanical and Aerospace Engineering 
Rights: © 2019 IEEE. All rights reserved.
Fulltext Permission: none
Fulltext Availability: No Fulltext
Appears in Collections:MAE Journal Articles

SCOPUSTM   
Citations 20

23
Updated on Mar 1, 2024

Web of ScienceTM
Citations 20

16
Updated on Oct 28, 2023

Page view(s)

90
Updated on Mar 4, 2024

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.