تقرير
FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
| العنوان: | FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds |
|---|---|
| المؤلفون: | Zhang, Yiming, Gu, Yicheng, Zeng, Yanhong, Xing, Zhening, Wang, Yuancheng, Wu, Zhizheng, Chen, Kai |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computer Vision and Pattern Recognition, Computer Science - Sound, Electrical Engineering and Systems Science - Audio and Speech Processing |
| الوصف: | We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. Despite its wide range of applications, existing approaches encounter limitations when it comes to simultaneously synthesizing high-quality and video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To overcome these limitations, we propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. FoleyCrafter comprises two key components: the semantic adapter for semantic alignment and the temporal controller for precise audio-video synchronization. The semantic adapter utilizes parallel cross-attention layers to condition audio generation on video features, producing realistic sound effects that are semantically relevant to the visual content. Meanwhile, the temporal controller incorporates an onset detector and a timestampbased adapter to achieve precise audio-video alignment. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents. We conduct extensive quantitative and qualitative experiments on standard benchmarks to verify the effectiveness of FoleyCrafter. Models and codes are available at https://github.com/open-mmlab/FoleyCrafterTest. Comment: Project page: https://foleycrafter.github.ioTest/ |
| نوع الوثيقة: | Working Paper |
| الوصول الحر: | http://arxiv.org/abs/2407.01494Test |
| رقم الانضمام: | edsarx.2407.01494 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |