دورية أكاديمية
DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptation ...
| العنوان: | DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptation ... |
|---|---|
| المؤلفون: | Xu, Jie, Saravanan, Karthikeyan, van Dalen, Rogier, Mehmood, Haaris, Tuckey, David, Ozay, Mete |
| بيانات النشر: | arXiv |
| سنة النشر: | 2024 |
| المجموعة: | DataCite Metadata Store (German National Library of Science and Technology) |
| مصطلحات موضوعية: | Machine Learning cs.LG, Cryptography and Security cs.CR, Distributed, Parallel, and Cluster Computing cs.DC, FOS Computer and information sciences |
| الوصف: | Federated learning (FL) allows clients in an Internet of Things (IoT) system to collaboratively train a global model without sharing their local data with a server. However, clients' contributions to the server can still leak sensitive information. Differential privacy (DP) addresses such leakage by providing formal privacy guarantees, with mechanisms that add randomness to the clients' contributions. The randomness makes it infeasible to train large transformer-based models, common in modern IoT systems. In this work, we empirically evaluate the practicality of fine-tuning large scale on-device transformer-based models with differential privacy in a federated learning system. We conduct comprehensive experiments on various system properties for tasks spanning a multitude of domains: speech recognition, computer vision (CV) and natural language understanding (NLU). Our results show that full fine-tuning under differentially private federated learning (DP-FL) generally leads to huge performance degradation ... : 16 pages, 10 figures, 5 tables ... |
| نوع الوثيقة: | article in journal/newspaper report |
| اللغة: | unknown |
| DOI: | 10.48550/arxiv.2405.06368 |
| الإتاحة: | https://doi.org/10.48550/arxiv.2405.06368Test https://arxiv.org/abs/2405.06368Test |
| حقوق: | Creative Commons Attribution Non Commercial No Derivatives 4.0 International ; https://creativecommons.org/licenses/by-nc-nd/4.0/legalcodeTest ; cc-by-nc-nd-4.0 |
| رقم الانضمام: | edsbas.C572559 |
| قاعدة البيانات: | BASE |
| DOI: | 10.48550/arxiv.2405.06368 |
|---|