Online Federated Learning over decentralized networks

Abstract

Online Federated Learning refers to the online optimization that is distributed over a decentralized network while still seek for training high-quality models. It allows each node to perform local operation and only contacts with its immediate neighbors, liberating it from the control of the `master' node. The computation and communication are totally decentralized, avoiding the traffic congestion and network coordination problems that are inevitable to most centralized distributed optimization. Our work addresses several critical problems and their corresponding solutions to make OFL more practical and more efficient in real-scenarios: (a) sampling technique to replace the costly deterministic communication cost with a stochastic strategy; (b) developing and analyzing an algorithm to seek for the optimal saddle point for decentralized online convex-concave problems, and therefore providing solutions for constrained decentralized optimization; (c) avoiding the challenges of “Pareto optimality” when the optimal values for models may only be similar rather than identical.