تقرير
Let There Be Sound: Reconstructing High Quality Speech from Silent Videos
| العنوان: | Let There Be Sound: Reconstructing High Quality Speech from Silent Videos |
|---|---|
| المؤلفون: | Kim, Ji-Hoon, Kim, Jaehun, Chung, Joon Son |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Electrical Engineering and Systems Science - Audio and Speech Processing, Computer Science - Artificial Intelligence, Computer Science - Machine Learning, Computer Science - Sound |
| الوصف: | The goal of this work is to reconstruct high quality speech from lip motions alone, a task also known as lip-to-speech. A key challenge of lip-to-speech systems is the one-to-many mapping caused by (1) the existence of homophenes and (2) multiple speech variations, resulting in a mispronounced and over-smoothed speech. In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives. Specifically, we incorporate (1) self-supervised speech representations to disambiguate homophenes, and (2) acoustic variance information to model diverse speech styles. Additionally, to better solve the aforementioned problem, we employ a flow based post-net which captures and refines the details of the generated speech. We perform extensive experiments on two datasets, and demonstrate that our method achieves the generation quality close to that of real human utterance, outperforming existing methods in terms of speech naturalness and intelligibility by a large margin. Synthesised samples are available at our demo page: https://mm.kaist.ac.kr/projects/LTBSTest. Comment: Accepted to AAAI 2024 |
| نوع الوثيقة: | Working Paper |
| الوصول الحر: | http://arxiv.org/abs/2308.15256Test |
| رقم الانضمام: | edsarx.2308.15256 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |