pFedKT: personalized federated learning with dual knowledge transfer

Abstract

Federated learning (FL) has been widely studied as an emerging privacy-preserving machine learning paradigm for achieving multi-party collaborative model training on decentralized data. In practice, such data tend to follow non-independent and identically distributed (non-IID) data distributions. Thus, the performance of models obtained through vanilla horizontal FL tends to vary significantly across FL clients. To tackle this challenge, a new subfield of FL – personalized federated learning (PFL) – has emerged for producing personalized FL models that can perform well on diverse local datasets. Existing PFL approaches are limited in terms of effectively transferring knowledge among clients to improve model generalization while achieving good performance on diverse local datasets. To bridge this important gap, we propose the personalized Federated Knowledge Transfer (pFedKT) approach. It involves dual knowledge transfer: (1) transferring historical local knowledge to local models via local hypernetworks; and (2) transferring latest global knowledge to local models through contrastive learning. By fusing historical local knowledge and the latest global knowledge, the personalization and generalization of individual models for FL clients can be simultaneously enhanced. We provide theoretical analysis on the generalization and convergence of pFedKT. Extensive experiments on 3 real-world datasets demonstrate that pFedKT achieves 0.74%–1.62% higher test accuracy compared to 14 state-of-the-art baselines.