Denoising based Sequence-to-Sequence Pre-training for Text Generation
| العنوان: | Denoising based Sequence-to-Sequence Pre-training for Text Generation |
|---|---|
| المؤلفون: | Ruoyu Jia, Liang Wang, Sujian Li, Jingming Liu, Wei Zhao |
| المصدر: | EMNLP/IJCNLP (1) |
| بيانات النشر: | Association for Computational Linguistics, 2019. |
| سنة النشر: | 2019 |
| مصطلحات موضوعية: | FOS: Computer and information sciences, Computer Science - Computation and Language, Computer science, business.industry, Noise reduction, Pattern recognition, 010501 environmental sciences, 01 natural sciences, Automatic summarization, 030507 speech-language pathology & audiology, 03 medical and health sciences, Text generation, Artificial intelligence, 0305 other medical science, business, Computation and Language (cs.CL), Encoder, 0105 earth and related environmental sciences, Transformer (machine learning model) |
| الوصف: | This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre-trains both the encoder and decoder by denoising the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and grammatical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence. Comment: Accepted to EMNLP 2019 |
| الوصول الحر: | https://explore.openaire.eu/search/publication?articleId=doi_dedup___::c780948c920c7e9a874adb642b28a614Test https://doi.org/10.18653/v1/d19-1412Test |
| حقوق: | OPEN |
| رقم الانضمام: | edsair.doi.dedup.....c780948c920c7e9a874adb642b28a614 |
| قاعدة البيانات: | OpenAIRE |
| الوصف غير متاح. |