Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0)
A description of the work "Crosslingual Generalization through Multitask Finetuning" by N. Muennighoff et al. which appeared on arxiv in November 2022. This work was produced as part of the BigScience Workshop.
Timestamps:
00:00 - Crosslingual Generalization through Multitask Finetuning (BLOOMZ & mT0)
01:56 - Finetuning data
03:55 - Dataset variants
05:11 - Finetuned models
06:46 - Evaluation details
07:35 - Results: Zero-shot multilingual task generalization
09:31 - Results: Language generalization
10:11 - Results: Language generalization (on "never intentionally seen" languages)
11:36 - Results: Multilingual prompting
12:44 - Results: Scaling multitask finetuning
13:31 - Influence of multitask finetuning on generative tasks
15:40 - Code generation
17:06 - Influence of language proportions
18:12 - Qualitative results
21:18 - Related Work
23:25 - Takeaways and released assets
24:24 - Nuts and bolts of finetuning
Detailed description:
This video describes studies "Crosslingual Generalization through Multitask Finetuning", a work that studies multilingual multitask finetuning for zero-shot task generalization on non-English tasks.
We start by describing the motivation for the work: previous studies have shown that finetuning large language models (LLMs) brings benefits, but has mostly focused on English models and tasks. While multilingual LLMs demonstrate promise, their zero-shot performance still trails their performance with task and language-specific finetuning. Collecting such finetuning is difficult for low-resource languages and tasks.
We summarise the key findings of this work:
(1) English multitask finetuning helps non-English tasks,
(2) multilingual finetuning data further helps,
(3) larger models benefit more from multitask finetuning,
(4) finetuning helps tasks on rarely seen languages.
We outline the finetuning data used in this work, xP3, which expands the P3 dataset used in prior work by adding 28 multilingual datasets. A second variant dubbed xP3mt is also employed for finetuning. This uses machine translation to generate non-English prompts from those found in xP3.
We then describe the families of models finetuned in the study: BLOOM and mT5, and how finetuning on xP3 is used to produce BLOOMZ and mT0. Next, we describe evaluation details - the three held-out task clusters (coreference resolution, sentence completion and natural language inference) and the additional assessment conducted on program synthesis (on HumanEval).
We then turn to results, first describing how mT0-13B outperforms BLOOMZ for zero-shot multilingual task generalization, then describing how xP3 finetuning yields better results than P3 finetuning (which in turn yields a boost even on non-English tasks). We describe how performance improves on languages "never intentionally seen" during pretraining, and how training on machine-translated prompts can boost performance on non-English prompts (though not always).
The video describes a scaling study that suggests that multitask finetuning benefits larger models more, and flags how multitask finetuning can harm generative task performance. This is illustrated both qualitatively and quantitatively with code generation. The influence of pretraining language proportions is also discussed.
Qualitative results are provided, together with a brief summary of related work. The video wraps up with a list of takeaways and released assets and some low-level nuts and bolts details about fine-tuning.
Answer to the question "Why is the sky blue?"
https://xkcd.com/1818/
Full author list:
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel
Note: as a co-author of this work, my description of this work may be favourably biased in its assessment.
Topics: #BLOOMZ #mT0 #bigscience #foundationmodels #languagemodels
Slides (pdf): https://samuelalbanie.com/files/diges...
References for papers mentioned in the video can be found at
http://samuelalbanie.com/digests/2022...
Arxiv paper: https://arxiv.org/abs/2211.01786
Code and models: https://github.com/bigscience-worksho...
BigScience Workshop: https://bigscience.huggingface.co/
Lastly, I'd like to express my thanks to the mystery donor of the Spanish voiceover. (Optional) If you'd like to support future videos: https://www.buymeacoffee.com/samuelal...