-
1تقرير
المؤلفون: Xu, Jie, Saravanan, Karthikeyan, van Dalen, Rogier, Mehmood, Haaris, Tuckey, David, Ozay, Mete
مصطلحات موضوعية: Computer Science - Machine Learning, Computer Science - Cryptography and Security, Computer Science - Distributed, Parallel, and Cluster Computing
الوصف: Federated learning (FL) allows clients in an Internet of Things (IoT) system to collaboratively train a global model without sharing their local data with a server. However, clients' contributions to the server can still leak sensitive information. Differential privacy (DP) addresses such leakage by providing formal privacy guarantees, with mechanisms that add randomness to the clients' contributions. The randomness makes it infeasible to train large transformer-based models, common in modern IoT systems. In this work, we empirically evaluate the practicality of fine-tuning large scale on-device transformer-based models with differential privacy in a federated learning system. We conduct comprehensive experiments on various system properties for tasks spanning a multitude of domains: speech recognition, computer vision (CV) and natural language understanding (NLU). Our results show that full fine-tuning under differentially private federated learning (DP-FL) generally leads to huge performance degradation which can be alleviated by reducing the dimensionality of contributions through parameter-efficient fine-tuning (PEFT). Our benchmarks of existing DP-PEFT methods show that DP-Low-Rank Adaptation (DP-LoRA) consistently outperforms other methods. An even more promising approach, DyLoRA, which makes the low rank variable, when naively combined with FL would straightforwardly break differential privacy. We therefore propose an adaptation method that can be combined with differential privacy and call it DP-DyLoRA. Finally, we are able to reduce the accuracy degradation and word error rate (WER) increase due to DP to less than 2% and 7% respectively with 1 million clients and a stringent privacy budget of {\epsilon}=2.
Comment: 16 pages, 10 figures, 5 tablesالوصول الحر: http://arxiv.org/abs/2405.06368Test
-
2تقرير
المؤلفون: Granqvist, Filip, Song, Congzheng, Cahill, Áine, van Dalen, Rogier, Pelikan, Martin, Chan, Yi Sheng, Feng, Xiaojun, Krishnaswami, Natarajan, Jina, Vojta, Chitnis, Mona
مصطلحات موضوعية: Computer Science - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Cryptography and Security, Computer Science - Computer Vision and Pattern Recognition
الوصف: Federated learning (FL) is an emerging machine learning (ML) training paradigm where clients own their data and collaborate to train a global model, without revealing any data to the server and other participants. Researchers commonly perform experiments in a simulation environment to quickly iterate on ideas. However, existing open-source tools do not offer the efficiency required to simulate FL on larger and more realistic FL datasets. We introduce pfl-research, a fast, modular, and easy-to-use Python framework for simulating FL. It supports TensorFlow, PyTorch, and non-neural network models, and is tightly integrated with state-of-the-art privacy algorithms. We study the speed of open-source FL frameworks and show that pfl-research is 7-72$\times$ faster than alternative open-source frameworks on common cross-device setups. Such speedup will significantly boost the productivity of the FL research community and enable testing hypotheses on realistic FL datasets that were previously too resource intensive. We release a suite of benchmarks that evaluates an algorithm's overall performance on a diverse set of realistic scenarios. The code is available on GitHub at https://github.com/apple/pfl-researchTest.
الوصول الحر: http://arxiv.org/abs/2404.06430Test
-
3تقرير
المؤلفون: van Dalen, Rogier
مصطلحات موضوعية: Electrical Engineering and Systems Science - Audio and Speech Processing, Computer Science - Machine Learning, Computer Science - Sound, 68T10
الوصف: The Transducer (e.g. RNN-Transducer or Conformer-Transducer) generates an output label sequence as it traverses the input sequence. It is straightforward to use in streaming mode, where it generates partial hypotheses before the complete input has been seen. This makes it popular in speech recognition. However, in streaming mode the Transducer has a mathematical flaw which, simply put, restricts the model's ability to change its mind. The fix is to replace local normalisation (e.g. a softmax) with global normalisation, but then the loss function becomes impossible to evaluate exactly. A recent paper proposes to solve this by approximating the model, severely degrading performance. Instead, this paper proposes to approximate the loss function, allowing global normalisation to apply to a state-of-the-art streaming model. Global normalisation reduces its word error rate by 9-11% relative, closing almost half the gap between streaming and lookahead mode.
Comment: 9 pages plus references and appendicesالوصول الحر: http://arxiv.org/abs/2307.10975Test
-
4تقرير
مصطلحات موضوعية: Computer Science - Computation and Language, Computer Science - Sound, Electrical Engineering and Systems Science - Audio and Speech Processing
الوصف: Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference as well as training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while lowering the training and inference times by up to 28$\%$ and reducing the memory budget by a factor of two. The benefits of SummaryMixing can also be generalized to other speech-processing tasks, such as speech understanding.
الوصول الحر: http://arxiv.org/abs/2307.07421Test
-
5تقرير
المؤلفون: Xu, Mingbin, Song, Congzheng, Tian, Ye, Agrawal, Neha, Granqvist, Filip, van Dalen, Rogier, Zhang, Xiao, Argueta, Arturo, Han, Shiyi, Deng, Yaqiao, Liu, Leo, Walia, Anmol, Jin, Alex
مصطلحات موضوعية: Computer Science - Machine Learning, Computer Science - Computation and Language, Computer Science - Cryptography and Security
الوصف: Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the DP-noise introduced to the model increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to decrease noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. This combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.
الوصول الحر: http://arxiv.org/abs/2207.08988Test
-
6تقرير
مصطلحات موضوعية: Computer Science - Cryptography and Security, Computer Science - Computation and Language, Computer Science - Machine Learning
الوصف: Federated learning with differential privacy, i.e. private federated learning (PFL), makes it possible to train models on private data distributed across users' devices without harming privacy. PFL is efficient for models, such as neural networks, that have a fixed number of parameters, and thus a fixed-dimensional gradient vector. Such models include neural-net language models, but not tokenizers, the topic of this work. Training a tokenizer requires frequencies of words from an unlimited vocabulary, and existing methods for finding an unlimited vocabulary need a separate privacy budget. A workaround is to train the tokenizer on publicly available data. However, in this paper we first show that a tokenizer trained on mismatched data results in worse model performance compared to a privacy-violating "oracle" tokenizer that accesses user data, with perplexity increasing by 20%. We also show that sub-word tokenizers are better suited to the federated context than word-level ones, since they can encode new words, though with more tokens per word. Second, we propose a novel method to obtain a tokenizer without using any additional privacy budget. During private federated learning of the language model, we sample from the model, train a new tokenizer on the sampled sequences, and update the model embeddings. We then continue private federated learning, and obtain performance within 1% of the "oracle" tokenizer. Since this process trains the tokenizer only indirectly on private data, we can use the "postprocessing guarantee" of differential privacy and thus use no additional privacy budget.
الوصول الحر: http://arxiv.org/abs/2203.09943Test
-
7تقريرEnforcing fairness in private federated learning via the modified method of differential multipliers
مصطلحات موضوعية: Computer Science - Machine Learning, Statistics - Machine Learning
الوصف: Federated learning with differential privacy, or private federated learning, provides a strategy to train machine learning models while respecting users' privacy. However, differential privacy can disproportionately degrade the performance of the models on under-represented groups, as these parts of the distribution are difficult to learn in the presence of noise. Existing approaches for enforcing fairness in machine learning models have considered the centralized setting, in which the algorithm has access to the users' data. This paper introduces an algorithm to enforce group fairness in private federated learning, where users' data does not leave their devices. First, the paper extends the modified method of differential multipliers to empirical risk minimization with fairness constraints, thus providing an algorithm to enforce fairness in the central setting. Then, this algorithm is extended to the private federated learning setting. The proposed algorithm, \texttt{FPFL}, is tested on a federated version of the Adult dataset and an "unfair" version of the FEMNIST dataset. The experiments on these datasets show how private federated learning accentuates unfairness in the trained models, and how FPFL is able to mitigate such unfairness.
Comment: Presented at PriML workshop at NeurIPS 2021. 20 pages: 11 of main content, 3 of references, and 6 of supplementary materialالوصول الحر: http://arxiv.org/abs/2109.08604Test
-
8تقرير
المؤلفون: Paulik, Matthias, Seigel, Matt, Mason, Henry, Telaar, Dominic, Kluivers, Joris, van Dalen, Rogier, Lau, Chi Wai, Carlson, Luke, Granqvist, Filip, Vandevelde, Chris, Agarwal, Sudeep, Freudiger, Julien, Byde, Andrew, Bhowmick, Abhishek, Kapoor, Gaurav, Beaumont, Si, Cahill, Áine, Hughes, Dominic, Javidbakht, Omid, Dong, Fei, Rishi, Rehan, Hung, Stanley
مصطلحات موضوعية: Computer Science - Machine Learning
الوصف: We describe the design of our federated task processing system. Originally, the system was created to support two specific federated tasks: evaluation and tuning of on-device ML systems, primarily for the purpose of personalizing these systems. In recent years, support for an additional federated task has been added: federated learning (FL) of deep neural networks. To our knowledge, only one other system has been described in literature that supports FL at scale. We include comparisons to that system to help discuss design decisions and attached trade-offs. Finally, we describe two specific large scale personalization use cases in detail to showcase the applicability of federated tuning to on-device personalization and to highlight application specific solutions.
Comment: 11 pages, 1 figureالوصول الحر: http://arxiv.org/abs/2102.08503Test
-
9تقرير
المؤلفون: Granqvist, Filip, Seigel, Matt, van Dalen, Rogier, Cahill, Áine, Shum, Stephen, Paulik, Matthias
مصطلحات موضوعية: Electrical Engineering and Systems Science - Audio and Speech Processing, Computer Science - Machine Learning, Computer Science - Sound, Statistics - Machine Learning
الوصف: Information on speaker characteristics can be useful as side information in improving speaker recognition accuracy. However, such information is often private. This paper investigates how privacy-preserving learning can improve a speaker verification system, by enabling the use of privacy-sensitive speaker data to train an auxiliary classification model that predicts vocal characteristics of speakers. In particular, this paper explores the utility achieved by approaches which combine different federated learning and differential privacy mechanisms. These approaches make it possible to train a central model while protecting user privacy, with users' data remaining on their devices. Furthermore, they make learning on a large population of speakers possible, ensuring good coverage of speaker characteristics when training a model. The auxiliary model described here uses features extracted from phrases which trigger a speaker verification system. From these features, the model predicts speaker characteristic labels considered useful as side information. The knowledge of the auxiliary model is distilled into a speaker verification system using multi-task learning, with the side information labels predicted by this auxiliary model being the additional task. This approach results in a 6% relative improvement in equal error rate over a baseline system.
Comment: To appear in proceedings of INTERSPEECH 2020الوصول الحر: http://arxiv.org/abs/2008.02651Test
-
10مؤتمر
المؤلفون: Parcollet, Titouan, Zhang, Shucong, van Dalen, Rogier, Ramos, Alberto Gil C. P., Bhattacharya, Sourav
المساهمون: Samsung AI Center Cambridge, Samsung AI
المصدر: Interspeech 2023 ; https://hal.science/hal-04116371Test ; Interspeech 2023, Aug 2023, Dublin, France
مصطلحات موضوعية: Efficient speech representation learning, SSL, [INFO]Computer Science [cs]
الوصف: International audience ; Speech representations learned with self-supervised learning (SSL) have the potential to significantly improve the performance of a number of audio applications, especially when availability of labeled data from the deployment domain is limited. Despite their successes, SSL training methods are compute-and memory-heavy, and require large investments in computing infrastructure, thus putting it out of the reach of most institutions. Therefore, building efficient model architectures is essential for the wide-scale adoption of SSL in speech technologies. CNN-based Acoustic Feature Extractors (AFE), which are widely used as encoders of acoustic waveforms, remain one of the main efficiency bottlenecks. This work proposes replacing CNN-based AFEs with more efficient ones and demonstrates that SSL pre-training time and memory consumption can be reduced by a factor of two to three over existing methods while preserving performances in speech-, command-, and speakerrecognition tasks.
العلاقة: hal-04116371; https://hal.science/hal-04116371Test; https://hal.science/hal-04116371/documentTest; https://hal.science/hal-04116371/file/_IS2023__On_the__In_Efficiency_of_Acoustic_Features_Extractors_for_Self_Supervised_Speech_Representation_Learning-2.pdfTest
