Federated fine-tuning of foundation ai models: lightweight computing approaches

Abstract

This research investigates the intersection of parameter-efficient fine-tuning (PEFT) methods and federated learning algorithms to address the computational, privacy, and performance challenges in deploying foundation AI models. Through systematic evaluation of four PEFT approaches (LoRA, P-Tuning, Prefix Tuning, and Discrete Prompt Tuning) across seven federated learning algorithms, we demonstrate taskspecific optimal combinations with significant efficiency gains. LoRA achieved superior performance on NLP tasks (84.87% accuracy with SCAFFOLD), while P-Tuning excelled in computer vision tasks (65.39% accuracy with FedAvg). Our experiments reveal that selective combinations reduce communication overhead by up to three orders of magnitude while maintaining competitive accuracy. We identify novel algorithmic synergies, such as LoRA-SCAFFOLD for language tasks and P-Tuning-FedAvg for vision tasks, and establish theoretical principles governing PEFT-federated algorithm interactions. These findings provide a comprehensive framework for efficiently deploying foundation models in resource-constrained, privacysensitive distributed environments, enabling wider accessibility of state-of-the-art AI capabilities across diverse computational landscapes. Code and models are publicly available at https://github.com/Tan-Yu/FYPFederated-Learning.