DeltaLM: Encoder Decoder Pre training for Language Generation and Translation
∆LM is a powerful pretrained multilingual encoder-decoder model for both language generation and translation. ∆LM initializes encoder and decoder with InfoXLM-base. It uses span corruption and translation span corruption for pretraining. ∆LM seems to perform better than mT5 and mBART on various language generation and translation benchmark datasets.
In this video, I will briefly provide an overview of the ∆LM architecture, pretraining recipe and its comparison with other encoder-decoder multilingual models for language generation and translation.
Here is the agenda:
00:00:00 What is DeltaLM (∆LM)?
00:02:16 What is the interleaved Transformer decoder in ∆LM?
00:04:12 What are pre-training tasks for ∆LM?
00:06:27 Results for Multilingual Language Generation
00:08:35 Results for Cross-lingual Language Generation
00:12:40 Results for Zero-shot cross-lingual abstractive text summarization
For more details, please look at https://arxiv.org/pdf/2106.13736.pdf
Ma, Shuming, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. "Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders." arXiv preprint arXiv:2106.13736 (2021).